diff --git a/docs/source/api.rst b/docs/source/api.rst index 2f6538a2..93ac4ce0 100644 --- a/docs/source/api.rst +++ b/docs/source/api.rst @@ -27,6 +27,7 @@ API Reference api/commit api/scan api/read + api/format_table api/predicate api/file_format api/file_system diff --git a/docs/source/api/format_table.rst b/docs/source/api/format_table.rst new file mode 100644 index 00000000..748b5ea1 --- /dev/null +++ b/docs/source/api/format_table.rst @@ -0,0 +1,33 @@ +.. Licensed to the Apache Software Foundation (ASF) under one +.. or more contributor license agreements. See the NOTICE file +.. distributed with this work for additional information +.. regarding copyright ownership. The ASF licenses this file +.. to you under the Apache License, Version 2.0 (the +.. "License"); you may not use this file except in compliance +.. with the License. You may obtain a copy of the License at + +.. http://www.apache.org/licenses/LICENSE-2.0 + +.. Unless required by applicable law or agreed to in writing, +.. software distributed under the License is distributed on an +.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +.. KIND, either express or implied. See the License for the +.. specific language governing permissions and limitations +.. under the License. + +============ +Format Table +============ + +.. _cpp-api-format-table: + +Table +===== + +.. doxygenclass:: paimon::FormatTable + :members: + :undoc-members: + +Reading and writing a format table go through the generic entry points every other table uses: +:cpp:class:`paimon::TableScan`, :cpp:class:`paimon::TableRead`, :cpp:class:`paimon::FileStoreWrite` +and :cpp:class:`paimon::FileStoreCommit`. See :doc:`../user_guide/format_table`. diff --git a/docs/source/user_guide.rst b/docs/source/user_guide.rst index 38ef0fc6..326f2480 100644 --- a/docs/source/user_guide.rst +++ b/docs/source/user_guide.rst @@ -33,6 +33,7 @@ User Guide user_guide/data_types user_guide/primary_key_table user_guide/append_only_table + user_guide/format_table user_guide/system_tables user_guide/write user_guide/commit diff --git a/docs/source/user_guide/format_table.rst b/docs/source/user_guide/format_table.rst new file mode 100644 index 00000000..d6cbf7ac --- /dev/null +++ b/docs/source/user_guide/format_table.rst @@ -0,0 +1,272 @@ +.. Licensed to the Apache Software Foundation (ASF) under one +.. or more contributor license agreements. See the NOTICE file +.. distributed with this work for additional information +.. regarding copyright ownership. The ASF licenses this file +.. to you under the Apache License, Version 2.0 (the +.. "License"); you may not use this file except in compliance +.. with the License. You may obtain a copy of the License at + +.. http://www.apache.org/licenses/LICENSE-2.0 + +.. Unless required by applicable law or agreed to in writing, +.. software distributed under the License is distributed on an +.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +.. KIND, either express or implied. See the License for the +.. specific language governing permissions and limitations +.. under the License. + +.. Ported from the Paimon documentation: +.. https://github.com/apache/paimon/blob/master/docs/docs/concepts/rest/tables.mdx + +.. default-domain:: cpp +.. highlight:: cpp + +Format Table +============ +A format table is a directory that holds multiple files of the same format. It carries no +snapshots and no manifests: the files in the directory are the table, so reading it lists +directories and writing to it adds files. A table is a format table when its ``type`` option is +``format-table``; ``file.format`` then names the format of every file in it, which here is +``parquet`` or ``orc``. + +A partitioned format table uses the standard Hive directory layout, and its partitions are +discovered from that layout rather than from metadata. By default a partition directory is named +``key=value``; setting ``format-table.partition-path-only-value`` names it by the value alone. + +Because a directory of plain files records no row identity, a format table only accepts inserts. +Reads still carry the leading ``_VALUE_KIND`` field every ``BatchReader`` promises, so an engine +that reads a batch by field index sees the same layout it does for a managed table; every row of a +format table is an insert. + +Reading and writing +------------------- +A format table is not served through :cpp:func:`Catalog::GetTable`, which describes a managed +table; use :cpp:func:`Catalog::GetFormatTable` instead. + +Reading and writing go through the entry points every other table uses: ``TableScan::Create``, +``TableRead::Create``, ``FileStoreWrite::Create`` and ``FileStoreCommit::Create``, each from its +usual context builder. Each reads the table's schema - from under the table path, or from the one +the context carries - and dispatches to a format table when its ``type`` says so, which is what +Java Paimon does through ``FormatTable.newReadBuilder()`` and ``newBatchWriteBuilder()``. + +``FormatTable`` is the only format-table type in the public API. The classes behind it - +``FormatTableScan``, ``FormatTableRead``, ``FormatTableWrite``, ``FormatTableCommit``, +``FormatDataSplit`` and ``FormatCommitMessage`` - are implementation details under ``src`` and are +named here only to describe what happens. A caller never needs them: a plan comes back as +``Plan``, a split as ``Split``, and a commit message as ``CommitMessage``. + +A catalog that can load a format table itself overrides ``Catalog::LoadFormatTable()``, the +protected hook :cpp:func:`Catalog::GetFormatTable` calls. The file system catalog uses it to say +its metadata lives under the table location, and the REST catalog to take the location and the +schema from one response instead of two that could disagree. A catalog that does not override it +still serves format tables, by reading the location and the schema through the virtuals every +catalog has. + +``TableScan::ListPartitions()`` lists the partitions a scan can see. A format table answers it by +listing directories; every other table type returns ``NotImplemented`` for now. + +Not all of the generic interfaces fit. ``FileStoreCommit`` is mostly about snapshots and manifests - +expiring them, rolling back to one, filtering by a commit identifier recorded in one - and a format +table keeps none of that state, so those calls are refused rather than quietly doing nothing. +``FileStoreWrite::Compact()`` is refused for the same reason, and both write and commit take batch +writes only, since there is no snapshot to record a commit identifier or a watermark in. +``TableScan`` takes a partition filter and a limit; a predicate or a bucket filter is refused. Java +refuses a predicate from ``FormatTableScan.withFilter`` too, but ``FormatReadBuilder.newScan()`` +splits one first and hands the partition half to the scan, so there a predicate over partition +columns still prunes directories. See the limits below. + +Options given at the call win over the ones the schema stored, as they do for every other table - +except ``type``, which is structural and is read from the schema alone, so one read or write +cannot decide what kind of table this is. The merged result is validated, not the schema's own +options, so an option a format table refuses - ``metastore.partitioned-table``, a file format +nothing here can read - is refused wherever it comes from rather than dropped in silence. + +A setting the format path cannot act on is refused by name rather than quietly dropped: + +* ``ReadContextBuilder::SetReadSchema()``. A projected read schema can rename a column, prune a + nested one and give it metadata of its own; a format table's projection is a list of top-level + names, so name the columns instead. +* ``WithStreamingMode()`` on a scan or a write, a global index result on a scan, and a real-time + context on a scan, a read or a write: a format table has no snapshots, no real-time store and no + index. +* ``WriteContextBuilder::WithWriteSchema()``, which names a subset of the columns to write. +* ``WriteContextBuilder::WithWriteId()``, which prefixes a postpone-bucket writer's files so one + compaction reader can put them back in order; a format table has no buckets. +* ``CommitContextBuilder::IgnoreEmptyCommit(false)``, ``UseRESTCatalogCommit(true)`` and + ``AppendCommitCheckConflict(true)``. Keeping an empty commit means writing a snapshot that adds + no files, a rest-catalog commit sends that snapshot to a catalog, and the conflict check reads + the manifests of concurrent commits - none of which exist here. Each is refused only when set + away from its default, so an ordinary commit is unaffected. +* A scan predicate or bucket filter, as above, and more than one partition filter: a scan descends + one directory layout, so it takes the values of a single partition rather than a set of them. + +What a data file is opened with is not one of the refusals. ``EnablePrefetch()``, the read-ahead +cache and its ``CacheConfig``, and the ``Cache`` a read carries all apply, because a format table +opens its files through the same component the managed table path opens its own with. What differs +between the two paths is which files there are and how a row is put back together, not how a file +is read. + +Some settings are not refused because they describe machinery the format path never reaches, and +refusing them would refuse the defaults: ``EnableMultiThreadRowToBatch()`` on a read, a write's +temporary directory and spill configuration, and ``WithIgnoreNumBucketCheck()`` and +``WithIgnorePreviousFiles()`` on a write. They have no effect here: a format read hands out the +batches parquet or orc already produced rather than assembling them from rows, a format write +buffers in memory and never spills, and a table with no buckets has no bucket count to check and +no previous files to read back. + +A write is two-phase, since a directory has no metadata to switch atomically: a file is written +into a ``_temporary`` directory beside where it will end up, under a hidden name of its own, and +only the commit renames it into place. That is the layout Java Paimon's +``RenamingTwoPhaseOutputStream`` stages under. The directory and the name are both hidden, the +convention a Hive-style directory uses for output that is not committed table data, and what a scan +of this table skips. The ``_temporary`` directory is shared with every other writer +of the same table and is left behind after a commit. + +A plan is in-memory only. ``FormatDataSplit`` has no serialized form - ``Split::Serialize()`` +refuses it - and neither has ``FormatCommitMessage``: a format table's plan has no cross-runtime +encoding, so plan, read and commit within one process. + +A ``FormatTableWrite`` and a ``FormatTableCommit`` are each driven by one thread, but separate +ones may fill and add to a table at once: each write stages its files under a uuid of its own, and +each commit publishes only the files its own messages name. Two *overwriting* commits over the +same directory race, since an overwrite clears what is committed there before publishing anything. +A ``FormatTableScan`` may be shared, since planning leaves it as it was. + +``TableRead::CreateCountReader()`` is not implemented for a format table, so counting its rows +means reading them. That is a gap here rather than something the layout forces: ``parquet`` and +``orc`` both record a row count in their own footer. + +A writer starts a new file once the one it is filling reaches ``target-file-row-num`` rows or +``target-file-size`` bytes. Both are checked between batches rather than between rows, because a +batch is the unit this API writes in, so a file may pass either target by up to one batch. Java +checks the row count on every row and the size every thousand rows, and its files therefore sit +closer to the target. + +Aborting a write +---------------- +``FormatTableWrite::Abort()`` removes the files the write staged. It is the one call still allowed +after ``PrepareCommit()``, and that is what it is for: a write dropped *before* preparing clears +its staged files from its own destructor, so only a commit that is prepared and then abandoned +needs it. + +Path containment is checked on the path text, which stops a ``..`` from leaving the table but not +a symbolic link pointing out of it - the same as Java's own local file system behaviour. + +``FormatTableCommit::Abort()`` does the same for the messages a commit was given. **Neither undoes +a commit that succeeded**: once a file has been renamed into place it is no longer staged, and +nothing here will take it back. Java's committer removes the published path as well as the staged +one, which matters there because a commit publishes file by file with nothing watching; here a +commit that fails part way takes its own published files back before it returns, so an abort is +left with the staged files alone. Both are best effort and never fail, so a warning in the log is +the only signal that a file could not be removed. + +Give ``FormatTableCommit`` only the messages this job's own writers produced. A message names a +staged file by path, and a commit can tell that the path belongs to this table, sits in the +partition the message declares, and is staged rather than already published - not whose staged file +it is. A well-formed message from somewhere else is published, or discarded by ``Abort()``, like +any other. + +Relationship to Java Paimon +--------------------------- +Java serves format tables from a Hive or REST catalog, which holds the schema. This implementation +also serves them from a file system catalog, which keeps the schema under the table directory - an +extension Java does not have. Only for such a table are the ``schema`` and ``branch`` directories +below the location treated as metadata rather than as data. + +A file system catalog keeps a table's schema in ``schema`` and its branches in ``branch`` below +the table location, so under ``format-table.partition-path-only-value`` the first partition value +may not be ``schema`` or ``branch``: the directory a write would use is the one holding the +table's own metadata. Such a write is refused, as is an overwrite naming that partition - which +would otherwise delete the schema. A table served from a REST or Hive catalog keeps its schema +elsewhere, so there these are ordinary partition values and are read and written like any other. + +Under that same layout a partition value may not start with ``_`` or ``.`` either, whichever +catalog serves the table: the value is the whole directory name, and a scan skips every hidden +name. Java writes such a directory and then cannot read it back; here the write is refused +instead. The one exception is the value standing for a null partition, ``partition.default-name``, +which the scan reads at a partition level by design. Under the ``key=value`` layout the question +does not arise, since the key in front of the value keeps the directory name visible. + +A few smaller differences come from this library's own conventions: + +* a write takes one partition per batch: the batch declares it through + ``RecordBatch::SetPartition()``, every row is checked against that declaration, and a batch + mixing partitions is refused. Java routes row by row, so one write call there may land in any + number of partitions; +* a write takes its partition from ``RecordBatch::SetPartition()`` rather than from the rows, so + the values arrive as text. They are still read into their column types and rendered back out + before anything is named after them - the round trip Java's writer makes when it renders a + partition out of the row it is writing, through the partition computer its + ``FileStorePathFactory`` holds. The table therefore decides the directory name and the commit + message, not the spelling the caller used: with ``partition.legacy-name`` on, its default, a + ``DATE`` partition is written as its day count whether the caller wrote ``19723`` or + ``2024-01-01``, and as ``YYYY-MM-DD`` when the option is off. A value that cannot be read into + its column type is refused. ``FormatTableCommit``'s static partition is *not* put through that + round trip and is used as given, which is what Java's ``FormatTableCommit.buildPartitionPath`` + does with it too; +* a commit message carries the partition its file belongs to, and a commit checks that it agrees + with the directory the file sits in. Java's message carries none and derives the partition from + the committer's target path, so the two cannot disagree there. A message here is a public struct + a caller may have built itself, so the value is checked rather than trusted; +* a projection that names the same column twice is rejected when the read is built. Java reads + such a column once per entry; +* a row limit is the caller's: ``FormatTableScan`` takes one so that a plan can drop splits it + cannot need, but ``FormatTableRead`` does not bound the reader it hands out, and the caller stops + calling ``NextBatch()`` once it has enough. Java wraps its reader in a ``LimitRecordReader``. + +Current limits +-------------- +Compared with Java Paimon, this implementation does not yet support: + +* the ``csv``, ``json``, ``text`` and ``mosaic`` file formats, leaving ``parquet`` and ``orc``. + All four are line-delimited text in Java, which shares one line-reading layer between them; + this library has no text file format at all, so the first of them to be added has to bring that + layer with it; +* cutting one large data file into byte ranges so that several readers share it. Java does this + only for its line-delimited text formats, which are the ones missing here; ``parquet`` and + ``orc`` each record where their own row groups and stripes begin, and a reader handed a byte + range of one would have to find that out for itself; +* ``metastore.partitioned-table``, which moves partition visibility into the catalog, and the + Hive partition sync that goes with it; +* partition filters beyond equality on partition values, where Java accepts a full predicate. + Partition discovery here also lists one directory level at a time and applies the filter to each + name, while Java turns a leading run of equality constraints into a path and starts listing + below it; a table with many partitions therefore costs more listings here than in Java; +* ``scan.ignore-corrupt-files`` and ``scan.ignore-lost-files``, which are not implemented: a + corrupt or missing data file fails the read rather than being skipped; +* ``dynamic-partition-overwrite``. An overwrite here always replaces the partitions the commit + actually writes to, which is what Java does under that option's default of ``true``. Java also + has the other mode: with it off, and always for an unpartitioned table, an overwrite empties + everything the table holds - so a statement whose query returns nothing still clears the table. + A commit with no messages therefore clears nothing here where Java would; +* ``format-table.commit-hive-sync-url``, which registers committed partitions with a Hive + metastore; +* column default values. Java replaces a null in a column whose schema field declares a default + with that default as it writes; here the null is written as it came; +* a table every one of whose columns is a partition column. Java projects the partition columns + out of what it writes, leaving files that carry nothing but a row count; here such a schema is + refused when the table is created and when it is opened; +* ``TIMESTAMP``, ``DECIMAL``, ``FLOAT`` and ``DOUBLE`` partition columns, which Java allows. This + is a restriction of the whole library rather than of format tables. The types that do work are + ``BOOLEAN``, ``TINYINT``, ``SMALLINT``, ``INT``, ``BIGINT``, ``STRING`` and ``DATE`` - the set + the managed table path reads and writes partitions in. Any other, ``BINARY`` among them, is + refused when the table is created and when it is opened, rather than at the first read or write: + validation asks by building the partition computer that does the round trip, so there is one + answer rather than a list of types that could fall out of step with it. + +``data-file.path-directory`` has no effect here, and none in Java either: Java's format table +writer builds its paths from the table root rather than from that directory. +``format-table.implementation`` is honoured by the engines rather than by the table - in Java +Spark it selects between Paimon's own implementation and the engine's ``FileTable`` - so it has +no meaning inside this library. + +Validation +---------- +A table Java can serve and this library cannot is refused at creation rather than accepted and +then found unopenable, whichever catalog it is created through. It can still reach a catalog +another way - written by Java, or by an older client - so the same checks run again when the +table is opened. + +Whitespace in a partition value is judged by ASCII rules here, while Java uses +``Character.isWhitespace``; a value made only of non-ASCII whitespace therefore lands in a +partition of its own rather than in the default one. diff --git a/include/paimon/api.h b/include/paimon/api.h index 81f236bc..2ba2c3c3 100644 --- a/include/paimon/api.h +++ b/include/paimon/api.h @@ -20,24 +20,25 @@ #pragma once -#include "paimon/commit_context.h" // IWYU pragma: export -#include "paimon/defs.h" // IWYU pragma: export -#include "paimon/factories/factory.h" // IWYU pragma: export -#include "paimon/file_store_commit.h" // IWYU pragma: export -#include "paimon/file_store_write.h" // IWYU pragma: export -#include "paimon/fs/file_system_factory.h" // IWYU pragma: export -#include "paimon/memory/memory_pool.h" // IWYU pragma: export -#include "paimon/predicate/predicate.h" // IWYU pragma: export -#include "paimon/read_context.h" // IWYU pragma: export -#include "paimon/reader/batch_reader.h" // IWYU pragma: export -#include "paimon/record_batch.h" // IWYU pragma: export -#include "paimon/result.h" // IWYU pragma: export -#include "paimon/scan_context.h" // IWYU pragma: export -#include "paimon/statistics_mode.h" // IWYU pragma: export -#include "paimon/status.h" // IWYU pragma: export -#include "paimon/table/source/table_read.h" // IWYU pragma: export -#include "paimon/table/source/table_scan.h" // IWYU pragma: export -#include "paimon/write_context.h" // IWYU pragma: export +#include "paimon/commit_context.h" // IWYU pragma: export +#include "paimon/defs.h" // IWYU pragma: export +#include "paimon/factories/factory.h" // IWYU pragma: export +#include "paimon/file_store_commit.h" // IWYU pragma: export +#include "paimon/file_store_write.h" // IWYU pragma: export +#include "paimon/fs/file_system_factory.h" // IWYU pragma: export +#include "paimon/memory/memory_pool.h" // IWYU pragma: export +#include "paimon/predicate/predicate.h" // IWYU pragma: export +#include "paimon/read_context.h" // IWYU pragma: export +#include "paimon/reader/batch_reader.h" // IWYU pragma: export +#include "paimon/record_batch.h" // IWYU pragma: export +#include "paimon/result.h" // IWYU pragma: export +#include "paimon/scan_context.h" // IWYU pragma: export +#include "paimon/statistics_mode.h" // IWYU pragma: export +#include "paimon/status.h" // IWYU pragma: export +#include "paimon/table/format/format_table.h" // IWYU pragma: export +#include "paimon/table/source/table_read.h" // IWYU pragma: export +#include "paimon/table/source/table_scan.h" // IWYU pragma: export +#include "paimon/write_context.h" // IWYU pragma: export // IWYU pragma: begin_exports #include "paimon/realtime/realtime_context.h" diff --git a/include/paimon/catalog/catalog.h b/include/paimon/catalog/catalog.h index 9213bde9..fdf783fe 100644 --- a/include/paimon/catalog/catalog.h +++ b/include/paimon/catalog/catalog.h @@ -40,6 +40,7 @@ namespace paimon { using Instant = std::variant; class Database; +class FormatTable; class Table; class View; class Schema; @@ -208,6 +209,33 @@ class PAIMON_EXPORT Catalog { /// snapshot id ascending, or an error status. virtual Result> ListSnapshots( const Identifier& identifier, const std::string& branch = "") const = 0; + + /// Gets a format table: a directory of data files laid out like a standard Hive table. + /// + /// A format table carries no snapshots and no manifests, so it is loaded through its own + /// method rather than `GetTable()`. Reading and writing it then go through the same + /// `TableScan`, `TableRead`, `FileStoreWrite` and `FileStoreCommit` entry points every other + /// table uses. + /// + /// @param identifier Identifier of the table to get. + /// @return A result containing the format table, or an error status if the table does not + /// exist or its `type` option is not `format-table`. + Result> GetFormatTable(const Identifier& identifier) const; + + protected: + /// Loads `identifier` as a format table, which is what `GetFormatTable()` hands back. + /// + /// The default reads the location and the schema through the virtuals above, which costs two + /// requests that can disagree, and treats every directory below the location as table + /// content. A catalog that can do better overrides it and answers the two questions the + /// default has to guess at: whether the location and the schema can be read in one round + /// trip, and whether it keeps this table's metadata under the table path. + /// + /// @param identifier Identifier of the table to load. + /// @return A result containing the format table, or an error status if the table does not + /// exist or its `type` option is not `format-table`. + virtual Result> LoadFormatTable( + const Identifier& identifier) const; }; } // namespace paimon diff --git a/include/paimon/defs.h b/include/paimon/defs.h index 8062d3d2..93b1dc77 100644 --- a/include/paimon/defs.h +++ b/include/paimon/defs.h @@ -118,10 +118,33 @@ struct PAIMON_EXPORT Options { /// "page-size" - Memory page size, default value 64 kb. static const char PAGE_SIZE[]; + /// "type" - Type of the table. Default value is "table" (a managed paimon table); + /// "format-table" declares a directory of plain data files laid out like a Hive table. + static const char TYPE[]; + /// "file.format" - Specify the message format of data files. /// Default value is parquet. static const char FILE_FORMAT[]; + /// "format-table.file.compression" - File compression of a format table. It is consulted + /// after "file.compression" and before the bare "compression" key an engine's own writer + /// reads; when none of the three is set, the format decides. + static const char FORMAT_TABLE_FILE_COMPRESSION[]; + + /// "format-table.partition-path-only-value" - Whether a format table names a partition + /// directory by its value alone ("2025/01/") instead of "key=value" + /// ("year=2025/month=01/"). Default false. + static const char FORMAT_TABLE_PARTITION_PATH_ONLY_VALUE[]; + + /// "metastore.partitioned-table" - Whether a table's partitions are registered with the + /// catalog, which then decides their visibility. Default false; paimon-cpp reads partitions + /// from the directory layout and rejects a table that sets it. + static const char METASTORE_PARTITIONED_TABLE[]; + + /// "file.suffix.include.compression" - Whether a data file's name carries the compression it + /// was written with, for a format that records it inside the file. Default false. + static const char FILE_SUFFIX_INCLUDE_COMPRESSION[]; + /// "file-system" - Specify the file system. /// Default value is local. static const char FILE_SYSTEM[]; diff --git a/include/paimon/table/format/format_table.h b/include/paimon/table/format/format_table.h new file mode 100644 index 00000000..452cc4be --- /dev/null +++ b/include/paimon/table/format/format_table.h @@ -0,0 +1,196 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#pragma once + +#include +#include +#include +#include + +#include "paimon/catalog/identifier.h" +#include "paimon/result.h" +#include "paimon/schema/schema.h" +#include "paimon/status.h" +#include "paimon/type_fwd.h" +#include "paimon/visibility.h" + +struct ArrowSchema; + +namespace paimon { + +class FileSystem; + +/// A table that is a directory of data files of one format, laid out like a standard Hive table. +/// +/// It carries no snapshots and no manifests: the files in the directory are the table, and a +/// partitioned table's partitions are the `key=value` directories below its location, or the +/// bare-value ones when `format-table.partition-path-only-value` asks for that layout. A table is +/// a format table when its `type` option is `format-table`; `file.format` then names the format of +/// every file in it, defaulting to `parquet`. +/// +/// Reads return the table's own columns, and writes only insert: there is nowhere to record that a +/// row was updated or deleted, so the `_VALUE_KIND` field every `BatchReader` carries is filled +/// with inserts throughout. +/// +/// This is the only format table type in the public API. Reading and writing go through the entry +/// points every other table uses - `TableScan`, `TableRead`, `FileStoreWrite` and +/// `FileStoreCommit` - each of which recognises a format table from the schema under the table +/// path, so a plan comes back as a `Plan`, a split as a `Split` and a commit message as a +/// `CommitMessage`. +/// +/// `docs/source/user_guide/format_table.rst` lists what is not supported yet, notably the `csv`, +/// `json`, `text` and `mosaic` file formats and `metastore.partitioned-table`. +class PAIMON_EXPORT FormatTable { + public: + /// Formats a format table's files can be in. + enum class Format { + PARQUET, + ORC, + }; + + /// Parses the `file.format` option, case-insensitively. A format this library has no reader + /// for is rejected by name, rather than failing later with a missing-format-factory error. + static Result ParseFormat(const std::string& file_format); + + /// The identifier of a format, as it appears in `file.format` and as a file extension. + static std::string FormatToString(Format format); + + /// Loads a format table from its directory, reading the schema stored under it. + /// + /// This needs a schema file under the table directory, which is what a table created through + /// `SchemaManager` or a file system catalog has. A table whose schema lives in a metastore + /// has none, and is loaded through `Catalog::GetFormatTable()` instead. + /// + /// @param file_system File system holding the table directory. + /// @param table_path Root path of the table, which is also its data location. + /// @param identifier Logical table identifier, used for naming and error messages. + /// @param dynamic_options Options given at the call, which win over the ones stored in the + /// schema, as they do for every other table type. + static Result> Create( + const std::shared_ptr& file_system, const std::string& table_path, + const Identifier& identifier, + const std::map& dynamic_options = {}); + + /// Builds a format table from a schema that is already loaded, for a caller that has one in + /// hand, such as a catalog that just created the table. + /// + /// @param location Directory the data files live in. It may not be empty: every path this + /// table reads or writes is checked against it, and an empty one is a prefix of + /// nothing. A trailing separator names the same directory as none. + /// @param location_carries_paimon_metadata See `LocationCarriesPaimonMetadata()`. Only the + /// caller knows: a file system catalog puts metadata there, a REST or Hive catalog + /// keeps it in the metastore. + /// @param dynamic_options Options given at the call, which win over the ones stored in the + /// schema. + static Result> Create( + const std::shared_ptr& file_system, const std::string& location, + const Identifier& identifier, const std::shared_ptr& schema, + bool location_carries_paimon_metadata = false, + const std::map& dynamic_options = {}); + + ~FormatTable(); + + /// Directory the data files live in. + const std::string& Location() const { + return location_; + } + + /// Format of every data file in the directory. + Format GetFormat() const { + return format_; + } + + /// Fields the table is partitioned by, in the order their directories nest. + const std::vector& PartitionKeys() const; + + /// Compression new data files are written with. It is resolved from `file.compression`, then + /// `format-table.file.compression`, then the bare `compression` key an engine's own writer + /// reads, then what the table's format writes by default. + const std::string& FileCompression() const { + return file_compression_; + } + + /// Directory name standing for a null partition value, from `partition.default-name`. + const std::string& PartitionDefaultName() const { + return partition_default_name_; + } + + /// Whether a partition directory is named by its value alone (`2025/01/`) instead of + /// `key=value` (`year=2025/month=01/`), from `format-table.partition-path-only-value`. The + /// value-only layout carries no field names, so only the nesting order of the table's + /// partition keys says which key a directory belongs to. + bool PartitionOnlyValueInPath() const { + return partition_only_value_in_path_; + } + + /// Table options: the ones stored in the schema, with any given at the call on top. + const std::map& Options() const { + return options_; + } + + /// A name to identify this table. + std::string Name() const { + return identifier_.GetTableName(); + } + + /// Full name of the table, database.tableName. + std::string FullName() const; + + /// Schema of the table, including its partition fields. + std::shared_ptr LatestSchema() const { + return schema_; + } + + /// Schema of the table as an arrow schema, including its partition fields. + Result> GetArrowSchema() const; + + /// File system holding the table directory. + std::shared_ptr GetFileSystem() const { + return file_system_; + } + + /// Whether this table's own metadata lives under its location, as told by whoever loaded it. + /// + /// Only then are the `schema` and `branch` directories below the location table metadata + /// rather than table content. For a table whose schema lives in a metastore they are data, + /// and are read and written like any other directory. + bool LocationCarriesPaimonMetadata() const { + return location_carries_paimon_metadata_; + } + + private: + FormatTable(const std::shared_ptr& file_system, const std::string& location, + const Identifier& identifier, const std::shared_ptr& schema, + const std::map& options, Format format, + const std::string& file_compression, const std::string& partition_default_name, + bool partition_only_value_in_path, bool location_carries_paimon_metadata); + + std::shared_ptr file_system_; + std::string location_; + Identifier identifier_; + std::shared_ptr schema_; + std::map options_; + Format format_; + std::string file_compression_; + std::string partition_default_name_; + bool partition_only_value_in_path_ = false; + bool location_carries_paimon_metadata_ = false; +}; + +} // namespace paimon diff --git a/include/paimon/table/source/split.h b/include/paimon/table/source/split.h index a968f02c..7f296fb9 100644 --- a/include/paimon/table/source/split.h +++ b/include/paimon/table/source/split.h @@ -34,9 +34,12 @@ namespace paimon { class MemoryPool; /// An input split for reading operation. Needed by most batch computation engines. Support -/// Serialize and Deserialize, compatible with java version. -/// This split can be either a `DataSplit` (for direct data file reads) or an `IndexedSplit` -/// (for reads leveraging global indexes). +/// Serialize and Deserialize. +/// +/// This split can be a `DataSplit` (for direct data file reads), an `IndexedSplit` (for reads +/// leveraging global indexes), or a `FormatDataSplit` (for a format table's plain data files). +/// Only the first two are serializable; a `FormatDataSplit` is in-memory only and `Serialize()` +/// refuses it. class PAIMON_EXPORT Split { public: virtual ~Split() = default; diff --git a/include/paimon/table/source/table_scan.h b/include/paimon/table/source/table_scan.h index c9b42915..bb611259 100644 --- a/include/paimon/table/source/table_scan.h +++ b/include/paimon/table/source/table_scan.h @@ -19,7 +19,10 @@ #pragma once +#include #include +#include +#include #include "paimon/result.h" #include "paimon/table/source/plan.h" @@ -44,5 +47,15 @@ class PAIMON_EXPORT TableScan { /// /// @return A Result containing a shared pointer to the created `Plan` or an error status. virtual Result> CreatePlan() = 0; + + /// Lists the partitions the scan can see, each as its partition values keyed by field name, + /// in a stable order. A table with no partition keys has none. + /// + /// Only a format table answers it today: its partitions are the directories a scan descends, + /// so listing them costs no more than planning. Every other table type returns + /// `NotImplemented`. + /// + /// @return A Result containing the partitions, or an error status. + virtual Result>> ListPartitions() const; }; } // namespace paimon diff --git a/src/paimon/CMakeLists.txt b/src/paimon/CMakeLists.txt index a9810424..9f619547 100644 --- a/src/paimon/CMakeLists.txt +++ b/src/paimon/CMakeLists.txt @@ -134,6 +134,7 @@ set(PAIMON_COMMON_SRCS common/predicate/starts_with.cpp common/reader/batch_reader.cpp common/reader/concat_batch_reader.cpp + common/reader/data_file_reader_factory.cpp common/reader/predicate_batch_reader.cpp common/reader/prefetch_file_batch_reader_impl.cpp common/reader/reader_utils.cpp @@ -394,6 +395,18 @@ set(PAIMON_CORE_SRCS core/stats/simple_stats_evolution.cpp core/table/table.cpp core/table/bucket_mode.cpp + core/table/format/format_file_listing.cpp + core/table/format/format_file_naming.cpp + core/table/format/format_path_validation.cpp + core/table/format/format_table.cpp + core/table/format/format_table_commit.cpp + core/table/format/format_table_file_store_commit.cpp + core/table/format/format_table_file_store_write.cpp + core/table/format/format_table_loader.cpp + core/table/format/format_table_read.cpp + core/table/format/format_table_scan.cpp + core/table/format/format_table_write.cpp + core/table/format/lazy_concat_batch_reader.cpp core/table/sink/commit_message.cpp core/table/sink/commit_message_impl.cpp core/table/sink/commit_message_serializer.cpp @@ -881,6 +894,10 @@ if(PAIMON_BUILD_TESTS) core/stats/simple_stats_test.cpp core/table/table_test.cpp core/table/bucket_mode_test.cpp + core/table/format/format_file_listing_test.cpp + core/table/format/format_file_naming_test.cpp + core/table/format/format_table_test.cpp + core/table/format/lazy_concat_batch_reader_test.cpp core/table/sink/commit_message_test.cpp core/table/sink/commit_message_impl_test.cpp core/table/source/fallback_data_split_test.cpp diff --git a/src/paimon/common/defs.cpp b/src/paimon/common/defs.cpp index 36b9c1c9..1b88e95b 100644 --- a/src/paimon/common/defs.cpp +++ b/src/paimon/common/defs.cpp @@ -36,7 +36,13 @@ const char Options::SEQUENCE_GROUP[] = "sequence-group"; const char Options::BUCKET[] = "bucket"; const char Options::BUCKET_KEY[] = "bucket-key"; +const char Options::TYPE[] = "type"; const char Options::FILE_FORMAT[] = "file.format"; +const char Options::FORMAT_TABLE_FILE_COMPRESSION[] = "format-table.file.compression"; +const char Options::FORMAT_TABLE_PARTITION_PATH_ONLY_VALUE[] = + "format-table.partition-path-only-value"; +const char Options::METASTORE_PARTITIONED_TABLE[] = "metastore.partitioned-table"; +const char Options::FILE_SUFFIX_INCLUDE_COMPRESSION[] = "file.suffix.include.compression"; const char Options::FILE_SYSTEM[] = "file-system"; const char Options::TARGET_FILE_SIZE[] = "target-file-size"; const char Options::TARGET_FILE_ROW_NUM[] = "target-file-row-num"; diff --git a/src/paimon/common/reader/data_file_reader_factory.cpp b/src/paimon/common/reader/data_file_reader_factory.cpp new file mode 100644 index 00000000..7c742c93 --- /dev/null +++ b/src/paimon/common/reader/data_file_reader_factory.cpp @@ -0,0 +1,92 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include "paimon/common/reader/data_file_reader_factory.h" + +#include + +#include "paimon/common/data/blob_defs.h" +#include "paimon/common/reader/delegating_prefetch_reader.h" +#include "paimon/common/reader/prefetch_file_batch_reader_impl.h" +#include "paimon/format/file_format.h" +#include "paimon/format/file_format_factory.h" +#include "paimon/format/read_hints.h" +#include "paimon/format/reader_builder.h" +#include "paimon/fs/file_system.h" + +namespace paimon { + +namespace { + +/// Formats the prefetching reader cannot drive: `blob` is read whole rather than in batches, and +/// `avro` is row-oriented, so neither has the batch boundaries it reads ahead to. +bool FormatSupportsPrefetch(const std::string& format_identifier) { + return format_identifier != "blob" && format_identifier != "avro"; +} + +} // namespace + +Result> DataFileReaderFactory::CreateReaderBuilder( + const std::string& format_identifier, const std::map& format_options, + const std::map& extra_format_options, + const DataFileReadOptions& read_options, const std::shared_ptr& pool) { + std::map options = format_options; + // The blob placeholder channels are internal: a table option must not enable them, only + // `extra_format_options` may. + BlobDefs::EraseInternalPlaceholderOptions(&options); + for (const auto& [key, value] : extra_format_options) { + options[key] = value; + } + PAIMON_ASSIGN_OR_RAISE(std::unique_ptr file_format, + FileFormatFactory::Get(format_identifier, options)); + PAIMON_ASSIGN_OR_RAISE(std::unique_ptr reader_builder, + file_format->CreateReaderBuilder(read_options.read_batch_size)); + reader_builder->WithMemoryPool(pool); + reader_builder->WithCache(read_options.cache); + // Runtime read state rather than more format options, so each format can adapt: parquet turns + // its own pre-buffering off when the shared read-ahead cache is reading ahead. + ReadHints read_hints; + read_hints.prefetch_enabled = read_options.prefetch_enabled; + read_hints.read_ahead_cache_enabled = read_options.read_ahead_cache_enabled; + reader_builder->WithReadHints(read_hints); + return reader_builder; +} + +Result> DataFileReaderFactory::Open( + const std::string& format_identifier, const std::string& file_path, int64_t file_size, + const ReaderBuilder* reader_builder, const DataFileReadOptions& read_options, + const std::shared_ptr& file_system, const std::shared_ptr& executor, + const std::shared_ptr& pool) { + if (read_options.prefetch_enabled && FormatSupportsPrefetch(format_identifier)) { + PAIMON_ASSIGN_OR_RAISE( + std::unique_ptr prefetch_reader, + PrefetchFileBatchReaderImpl::Create( + file_path, file_size, reader_builder, file_system, + read_options.prefetch_max_parallel_num, read_options.read_batch_size, + read_options.prefetch_batch_count, read_options.adaptive_prefetch_strategy, + executor, + /*initialize_read_ranges=*/false, read_options.read_ahead_cache_enabled, + read_options.cache_config, pool)); + return std::make_unique(std::move(prefetch_reader)); + } + PAIMON_ASSIGN_OR_RAISE(std::shared_ptr input_stream, + file_system->Open(FileStatus(file_path, file_size))); + return reader_builder->Build(input_stream); +} + +} // namespace paimon diff --git a/src/paimon/common/reader/data_file_reader_factory.h b/src/paimon/common/reader/data_file_reader_factory.h new file mode 100644 index 00000000..7341cdac --- /dev/null +++ b/src/paimon/common/reader/data_file_reader_factory.h @@ -0,0 +1,91 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#pragma once + +#include +#include +#include +#include + +#include "paimon/read_context.h" +#include "paimon/result.h" + +namespace paimon { + +class Cache; +class Executor; +class FileBatchReader; +class FileSystem; +class MemoryPool; +class ReaderBuilder; + +/// Settings applied when a data file is opened, gathered from the read context and the table +/// options. The defaults are a plain read: no cache, no prefetch. +struct DataFileReadOptions { + /// Block cache the format reader may put what it reads into, or null for none. + std::shared_ptr cache; + /// Rows a batch holds. + int32_t read_batch_size = 0; + /// Whether files are read ahead of the batches being asked for. + bool prefetch_enabled = false; + uint32_t prefetch_max_parallel_num = 0; + uint32_t prefetch_batch_count = 0; + bool adaptive_prefetch_strategy = false; + /// Whether the shared read-ahead cache takes over reading ahead from the format itself. + bool read_ahead_cache_enabled = true; + CacheConfig cache_config; +}; + +/// Opens one data file as a `FileBatchReader`. +/// +/// Both table paths come through here, so a file is opened the same way whichever of them reached +/// it: a managed table through `AbstractSplitRead`, a format table through `FormatTableRead`. Each +/// arrives with its own files and its own schema, but which format reads a file, what it may +/// cache and whether it is read ahead are decided in one place rather than two that can drift. +class DataFileReaderFactory { + public: + DataFileReaderFactory() = delete; + ~DataFileReaderFactory() = delete; + + /// Builds the reader builder a format reads with. It is kept out of `Open()` so that a split + /// builds one and reuses it for every file. + /// + /// @param format_options The table's options, which the format reads its own settings from. + /// Internal blob placeholder options are dropped: only the internal read path may + /// enable those, through `extra_format_options`. + /// @param extra_format_options Options only the caller knows, applied over `format_options`. + static Result> CreateReaderBuilder( + const std::string& format_identifier, + const std::map& format_options, + const std::map& extra_format_options, + const DataFileReadOptions& read_options, const std::shared_ptr& pool); + + /// Opens `file_path`, reading ahead when the options ask for it and the format can. + /// + /// @param file_size Size of the file in bytes. It is trusted: an object-store read is issued + /// against it, so a caller holding a size it does not vouch for should check it against + /// the file system first. + static Result> Open( + const std::string& format_identifier, const std::string& file_path, int64_t file_size, + const ReaderBuilder* reader_builder, const DataFileReadOptions& read_options, + const std::shared_ptr& file_system, const std::shared_ptr& executor, + const std::shared_ptr& pool); +}; + +} // namespace paimon diff --git a/src/paimon/common/utils/hadoop_compression.h b/src/paimon/common/utils/hadoop_compression.h new file mode 100644 index 00000000..c56558dc --- /dev/null +++ b/src/paimon/common/utils/hadoop_compression.h @@ -0,0 +1,121 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +#pragma once + +#include +#include +#include + +#include "paimon/common/utils/string_utils.h" + +namespace paimon { + +/// The compressions hadoop names in a file's extension. +/// +/// A data file's name carries its compression when it is one of these - +/// `data--0.snappy.parquet` - and the option's own text otherwise. +/// +/// Every name and extension below is fixed by what the rest of the paimon ecosystem writes and +/// reads, not chosen here: `zstd` is spelled `zst` in a file name because that is the extension +/// those files carry. Changing one makes files written elsewhere unreadable, and files written +/// here unreadable elsewhere. +class HadoopCompression { + public: + enum class Kind { + NONE, + GZIP, + BZIP2, + DEFLATE, + SNAPPY, + LZ4, + ZSTD, + }; + + HadoopCompression() = delete; + ~HadoopCompression() = delete; + + /// The compression a `file.compression` value names, case-insensitively. An empty value and + /// "none" are `NONE`; a value naming no compression at all is nullopt, and a caller writes + /// that one into the file name verbatim. Only the names listed here are recognised, so + /// "uncompressed" is not one of them. + static std::optional FromName(const std::string& name) { + std::string normalized = StringUtils::ToLowerCase(name); + if (normalized.empty() || normalized == "none") { + return Kind::NONE; + } + for (const Kind kind : AllCompressions()) { + if (normalized == ToName(kind)) { + return kind; + } + } + return std::nullopt; + } + + /// The extension the compression adds to a file name, without its dot. Empty for `NONE`. + static std::string ToFileExtension(Kind kind) { + switch (kind) { + case Kind::GZIP: + return "gz"; + case Kind::BZIP2: + return "bz2"; + case Kind::DEFLATE: + return "deflate"; + case Kind::SNAPPY: + return "snappy"; + case Kind::LZ4: + return "lz4"; + case Kind::ZSTD: + return "zst"; + case Kind::NONE: + break; + } + return std::string(); + } + + private: + /// The name a `file.compression` value carries for the compression. + static std::string ToName(Kind kind) { + switch (kind) { + case Kind::GZIP: + return "gzip"; + case Kind::BZIP2: + return "bzip2"; + case Kind::DEFLATE: + return "deflate"; + case Kind::SNAPPY: + return "snappy"; + case Kind::LZ4: + return "lz4"; + case Kind::ZSTD: + return "zstd"; + case Kind::NONE: + break; + } + return std::string(); + } + + static const std::array& AllCompressions() { + static const std::array kAll = {Kind::GZIP, Kind::BZIP2, Kind::DEFLATE, + Kind::SNAPPY, Kind::LZ4, Kind::ZSTD}; + return kAll; + } +}; + +} // namespace paimon diff --git a/src/paimon/common/utils/string_utils.cpp b/src/paimon/common/utils/string_utils.cpp index 5b405895..c25b4151 100644 --- a/src/paimon/common/utils/string_utils.cpp +++ b/src/paimon/common/utils/string_utils.cpp @@ -61,7 +61,7 @@ bool StringUtils::EndsWith(const std::string& str, const std::string& suffix) { size_t s2 = suffix.size(); return (s1 >= s2) && (str.compare(s1 - s2, s2, suffix) == 0); } -bool StringUtils::IsNullOrWhitespaceOnly(const std::string& str) { +bool StringUtils::IsNullOrWhitespaceOnly(std::string_view str) { if (str.empty()) { return true; } diff --git a/src/paimon/common/utils/string_utils.h b/src/paimon/common/utils/string_utils.h index 3c0906e2..294a434e 100644 --- a/src/paimon/common/utils/string_utils.h +++ b/src/paimon/common/utils/string_utils.h @@ -31,6 +31,7 @@ #include #include #include +#include #include #include @@ -108,7 +109,9 @@ class PAIMON_EXPORT StringUtils { static bool EndsWith(const std::string& str, const std::string& suffix); - static bool IsNullOrWhitespaceOnly(const std::string& str); + /// Whether `str` is empty or holds nothing but whitespace, by ASCII rules. Takes a view so + /// that a caller checking one row of a column at a time does not allocate. + static bool IsNullOrWhitespaceOnly(std::string_view str); static void Trim(std::string* str); diff --git a/src/paimon/core/catalog/catalog.cpp b/src/paimon/core/catalog/catalog.cpp index 97f13afa..c247b02e 100644 --- a/src/paimon/core/catalog/catalog.cpp +++ b/src/paimon/core/catalog/catalog.cpp @@ -20,10 +20,14 @@ #include +#include "fmt/format.h" #include "paimon/catalog_options.h" #include "paimon/common/utils/string_utils.h" +#include "paimon/core/catalog/catalog_utils.h" #include "paimon/core/catalog/file_system_catalog.h" #include "paimon/core/core_options.h" +#include "paimon/schema/schema.h" +#include "paimon/table/format/format_table.h" #ifdef PAIMON_ENABLE_REST #include "paimon/rest/rest_catalog.h" #endif @@ -60,4 +64,15 @@ Result> Catalog::Create(const std::string& root_path, return std::make_unique(core_options.GetFileSystem(), root_path, options); } +Result> Catalog::GetFormatTable(const Identifier& identifier) const { + return LoadFormatTable(identifier); +} + +Result> Catalog::LoadFormatTable(const Identifier& identifier) const { + // The location and the schema are read separately, and every directory below the location is + // table content: a catalog that knows better overrides this and says so. + return CatalogUtils::LoadFormatTableInTwoRequests(*this, identifier, + /*metadata_under_table_path=*/false); +} + } // namespace paimon diff --git a/src/paimon/core/catalog/catalog_utils.cpp b/src/paimon/core/catalog/catalog_utils.cpp index 7a6768ab..0d939857 100644 --- a/src/paimon/core/catalog/catalog_utils.cpp +++ b/src/paimon/core/catalog/catalog_utils.cpp @@ -16,11 +16,18 @@ #include "paimon/core/catalog/catalog_utils.h" +#include +#include #include +#include #include "fmt/format.h" #include "paimon/catalog/catalog.h" +#include "paimon/core/options/table_type.h" +#include "paimon/defs.h" #include "paimon/result.h" +#include "paimon/schema/schema.h" +#include "paimon/table/format/format_table.h" namespace paimon { @@ -68,4 +75,46 @@ Status CatalogUtils::CheckNotBranch(const Identifier& identifier, const std::str return Status::OK(); } +Status CatalogUtils::CheckManagedTableType(const Identifier& identifier, + const std::shared_ptr& schema, + const std::string& action) { + std::shared_ptr data_schema = std::dynamic_pointer_cast(schema); + if (data_schema == nullptr) { + // Only a data table carries table options, so nothing else declares a table type. + return Status::OK(); + } + PAIMON_ASSIGN_OR_RAISE(TableType table_type, + TableTypeDefine::FromOptions(data_schema->Options())); + if (table_type == TableType::FORMAT_TABLE) { + return Status::Invalid( + fmt::format("Cannot open format table '{}' as a Table in '{}', please use " + "'Catalog::GetFormatTable' or 'FormatTable::Create'.", + identifier.ToString(), action)); + } + // A materialized table is a managed table that also carries the SQL it materializes. + if (table_type != TableType::TABLE && table_type != TableType::MATERIALIZED_TABLE) { + const std::map& options = data_schema->Options(); + auto type_iter = options.find(Options::TYPE); + return Status::NotImplemented(fmt::format( + "Cannot open table '{}' in '{}': its '{}' is '{}', a table type paimon-cpp does not " + "implement, and a managed table would promise snapshots it never had.", + identifier.ToString(), action, Options::TYPE, + type_iter == options.end() ? std::string() : type_iter->second)); + } + return Status::OK(); +} + +Result> CatalogUtils::LoadFormatTableInTwoRequests( + const Catalog& catalog, const Identifier& identifier, bool metadata_under_table_path) { + PAIMON_ASSIGN_OR_RAISE(std::string location, catalog.GetTableLocation(identifier)); + PAIMON_ASSIGN_OR_RAISE(std::shared_ptr schema, catalog.LoadTableSchema(identifier)); + std::shared_ptr data_schema = std::dynamic_pointer_cast(schema); + if (data_schema == nullptr) { + return Status::Invalid(fmt::format("{} is not a data table, so it cannot be a format table", + identifier.GetFullName())); + } + return FormatTable::Create(catalog.GetFileSystem(), location, identifier, data_schema, + metadata_under_table_path); +} + } // namespace paimon diff --git a/src/paimon/core/catalog/catalog_utils.h b/src/paimon/core/catalog/catalog_utils.h index b9a54ac1..2cca3a20 100644 --- a/src/paimon/core/catalog/catalog_utils.h +++ b/src/paimon/core/catalog/catalog_utils.h @@ -16,13 +16,19 @@ #pragma once +#include #include #include "paimon/catalog/identifier.h" +#include "paimon/result.h" #include "paimon/status.h" namespace paimon { +class Catalog; +class FormatTable; +class Schema; + /// Checks shared by the catalog implementations. Every check takes an `action` naming /// the rejected operation in the error message, e.g. "dropTable". class CatalogUtils { @@ -41,6 +47,31 @@ class CatalogUtils { /// Fails when `identifier` carries a "$branch_" suffix. static Status CheckNotBranch(const Identifier& identifier, const std::string& action); + + /// Fails when `schema` is not a table a `Table` can describe. + /// + /// Only a managed table is one: a format table is loaded through `GetFormatTable()`, and the + /// remaining table types are stored differently and are not implemented here, so handing one + /// back as a `Table` would promise snapshots it never had. A schema that is not a data + /// table's, such as a system table's, passes. + /// + /// @param action The call being refused, qualified as a caller would write it, e.g. + /// `Catalog::GetTable`. It names the entry point in the message. + static Status CheckManagedTableType(const Identifier& identifier, + const std::shared_ptr& schema, + const std::string& action); + + /// Loads `identifier` as a format table by reading its location and its schema separately. + /// + /// What every catalog can do, so it is what `Catalog::GetFormatTable()` falls back to. A + /// catalog that can get both from one response should not use it, since two requests can + /// disagree. + /// + /// @param metadata_under_table_path Whether this catalog put the table's own metadata under + /// the table path, which is what tells a `schema` or `branch` directory below the + /// location from a partition directory of the same name. + static Result> LoadFormatTableInTwoRequests( + const Catalog& catalog, const Identifier& identifier, bool metadata_under_table_path); }; } // namespace paimon diff --git a/src/paimon/core/catalog/file_system_catalog.cpp b/src/paimon/core/catalog/file_system_catalog.cpp index 85907c12..06712247 100644 --- a/src/paimon/core/catalog/file_system_catalog.cpp +++ b/src/paimon/core/catalog/file_system_catalog.cpp @@ -327,6 +327,7 @@ Result> FileSystemCatalog::GetTable(const Identifier& ide identifier.GetTableName()); } PAIMON_ASSIGN_OR_RAISE(std::string table_path, GetTableLocation(identifier)); + // `Table::Create` reads the same schema file and rejects a format table there. return Table::Create(fs_, table_path, identifier); } @@ -529,4 +530,10 @@ Result> FileSystemCatalog::ListSnapshots( return result; } +Result> FileSystemCatalog::LoadFormatTable( + const Identifier& identifier) const { + return CatalogUtils::LoadFormatTableInTwoRequests(*this, identifier, + /*metadata_under_table_path=*/true); +} + } // namespace paimon diff --git a/src/paimon/core/catalog/file_system_catalog.h b/src/paimon/core/catalog/file_system_catalog.h index 3925aff8..155c74ba 100644 --- a/src/paimon/core/catalog/file_system_catalog.h +++ b/src/paimon/core/catalog/file_system_catalog.h @@ -69,6 +69,13 @@ class FileSystemCatalog : public Catalog { Result> ListSnapshots(const Identifier& identifier, const std::string& branch) const override; + protected: + /// This catalog keeps a table's schema under the table's own path, so the `schema` and + /// `branch` directories there are metadata rather than table content. Nothing else differs, + /// so the location and the schema are still read separately. + Result> LoadFormatTable( + const Identifier& identifier) const override; + private: static std::string NewDatabasePath(const std::string& warehouse, const std::string& db_name); static Result NewDataTablePath(const std::string& warehouse, diff --git a/src/paimon/core/catalog/file_system_catalog_test.cpp b/src/paimon/core/catalog/file_system_catalog_test.cpp index 9f6a568f..bfc23dcf 100644 --- a/src/paimon/core/catalog/file_system_catalog_test.cpp +++ b/src/paimon/core/catalog/file_system_catalog_test.cpp @@ -36,6 +36,7 @@ #include "paimon/fs/file_system.h" #include "paimon/fs/file_system_factory.h" #include "paimon/snapshot/snapshot_info.h" +#include "paimon/table/format/format_table.h" #include "paimon/testing/utils/testharness.h" namespace paimon::test { @@ -529,6 +530,41 @@ TEST(FileSystemCatalogTest, TestCreateTableWithBlob) { ArrowSchemaRelease(&schema); } +TEST(FileSystemCatalogTest, TestGetTableRejectsFormatTable) { + std::map options; + options[Options::FILE_SYSTEM] = "local"; + options[Options::FILE_FORMAT] = "orc"; + ASSERT_OK_AND_ASSIGN(auto core_options, CoreOptions::FromMap(options)); + auto dir = UniqueTestDirectory::Create(); + ASSERT_TRUE(dir); + FileSystemCatalog catalog(core_options.GetFileSystem(), dir->Str(), options); + ASSERT_OK(catalog.CreateDatabase("db1", options, /*ignore_if_exists=*/false)); + + arrow::Schema typed_schema( + {arrow::field("id", arrow::int32()), arrow::field("name", arrow::utf8())}); + ::ArrowSchema schema; + ASSERT_TRUE(arrow::ExportSchema(typed_schema, &schema).ok()); + std::map table_options = {{Options::TYPE, "format-table"}, + {Options::FILE_FORMAT, "parquet"}}; + ASSERT_OK(catalog.CreateTable(Identifier("db1", "fmt"), &schema, /*partition_keys=*/{}, + /*primary_keys=*/{}, table_options, + /*ignore_if_exists=*/false)); + ArrowSchemaRelease(&schema); + + // A format table has no snapshots and no manifests, so handing it back as a Table would + // describe it as something it is not. + ASSERT_NOK_WITH_MSG(catalog.GetTable(Identifier("db1", "fmt")), "Cannot open format table"); + ASSERT_OK_AND_ASSIGN(std::shared_ptr format_table, + catalog.GetFormatTable(Identifier("db1", "fmt"))); + ASSERT_EQ(format_table->GetFormat(), FormatTable::Format::PARQUET); + ASSERT_EQ(format_table->FullName(), "db1.fmt"); + + // This catalog keeps a table's schema under the table's own path, so the `schema` and + // `branch` directories there are its metadata and a scan leaves them alone. A catalog that + // keeps schemas elsewhere says no, and then every directory below the location is data. + ASSERT_TRUE(format_table->LocationCarriesPaimonMetadata()); +} + TEST(FileSystemCatalogTest, TestInvalidCreateTable) { std::map options; options[Options::FILE_SYSTEM] = "local"; diff --git a/src/paimon/core/core_options.cpp b/src/paimon/core/core_options.cpp index 6320ec57..e1338f62 100644 --- a/src/paimon/core/core_options.cpp +++ b/src/paimon/core/core_options.cpp @@ -377,6 +377,10 @@ class ConfigParser { // Impl is a private implementation of CoreOptions, // storing various configurable fields and their default values. +// The fields are grouped by the option each one parses, so that a new option lands beside the +// code that reads it. Ordering them by size instead would save a few dozen bytes in the one +// instance a table holds and scatter that grouping across the struct. +// NOLINTNEXTLINE(clang-analyzer-optin.performance.Padding) struct CoreOptions::Impl { int64_t page_size = 64 * 1024; std::optional target_file_size; @@ -487,6 +491,9 @@ struct CoreOptions::Impl { bool blob_as_descriptor = false; std::optional blob_split_by_file_size; bool legacy_partition_name_enabled = true; + bool file_suffix_include_compression = false; + bool format_table_partition_only_value_in_path = false; + bool metastore_partitioned_table = false; bool global_index_enabled = true; std::optional global_index_thread_num; bool commit_force_compact = false; @@ -645,6 +652,15 @@ struct CoreOptions::Impl { PAIMON_RETURN_NOT_OK(parser.ParseFileFormatPerLevel(&file_format_per_level)); // Parse file.compression.per.level - different compression for different levels PAIMON_RETURN_NOT_OK(parser.ParseFileCompressionPerLevel(&file_compression_per_level)); + // Parse file.suffix.include.compression - carry the compression in a data file's name + PAIMON_RETURN_NOT_OK(parser.Parse(Options::FILE_SUFFIX_INCLUDE_COMPRESSION, + &file_suffix_include_compression)); + // Parse format-table.partition-path-only-value - name a partition directory by its value + PAIMON_RETURN_NOT_OK(parser.Parse(Options::FORMAT_TABLE_PARTITION_PATH_ONLY_VALUE, + &format_table_partition_only_value_in_path)); + // Parse metastore.partitioned-table - partitions are registered with the catalog + PAIMON_RETURN_NOT_OK( + parser.Parse(Options::METASTORE_PARTITIONED_TABLE, &metastore_partitioned_table)); return Status::OK(); } @@ -1761,6 +1777,36 @@ bool CoreOptions::LegacyPartitionNameEnabled() const { return impl_->legacy_partition_name_enabled; } +bool CoreOptions::FileSuffixIncludeCompression() const { + return impl_->file_suffix_include_compression; +} + +bool CoreOptions::FormatTablePartitionOnlyValueInPath() const { + return impl_->format_table_partition_only_value_in_path; +} + +bool CoreOptions::MetastorePartitionedTable() const { + return impl_->metastore_partitioned_table; +} + +std::string CoreOptions::FormatTableFileCompression() const { + // The resolution order the rest of the paimon ecosystem follows; both the compression suffix + // in a file's name and its contents derive from it, so they cannot disagree. `compression` is + // not a paimon option of its own but the key an engine's own writer reads: paimon-spark copies + // `format-table.file.compression` onto it, so a table written that way carries only that key. + const char* const keys[] = {Options::FILE_COMPRESSION, Options::FORMAT_TABLE_FILE_COMPRESSION, + "compression"}; + for (const char* key : keys) { + auto iter = impl_->raw_options.find(key); + if (iter != impl_->raw_options.end()) { + return iter->second; + } + } + // What the format writes by default, when no option names a compression. + return impl_->file_format != nullptr && impl_->file_format->Identifier() == "parquet" ? "snappy" + : "zstd"; +} + bool CoreOptions::GlobalIndexEnabled() const { return impl_->global_index_enabled; } diff --git a/src/paimon/core/core_options.h b/src/paimon/core/core_options.h index 85a4a7fd..84f008d9 100644 --- a/src/paimon/core/core_options.h +++ b/src/paimon/core/core_options.h @@ -238,6 +238,23 @@ class PAIMON_EXPORT CoreOptions { bool LegacyPartitionNameEnabled() const; + /// Whether a data file's name carries the compression it was written with, from + /// `file.suffix.include.compression`. + bool FileSuffixIncludeCompression() const; + + /// Whether a format table names a partition directory by its value alone (`2025/01/`) instead + /// of `key=value` (`year=2025/month=01/`), from `format-table.partition-path-only-value`. + bool FormatTablePartitionOnlyValueInPath() const; + + /// Whether the table's partitions are registered with the catalog rather than discovered from + /// the directory layout, from `metastore.partitioned-table`. + bool MetastorePartitionedTable() const; + + /// Compression the data files of a format table are written with. It is resolved from + /// `file.compression`, then `format-table.file.compression`, then the bare `compression` key + /// an engine's own writer reads, then what the table's format writes by default. + std::string FormatTableFileCompression() const; + bool GlobalIndexEnabled() const; Result> CreateGlobalIndexExternalPath() const; diff --git a/src/paimon/core/core_options_test.cpp b/src/paimon/core/core_options_test.cpp index c110623d..49452216 100644 --- a/src/paimon/core/core_options_test.cpp +++ b/src/paimon/core/core_options_test.cpp @@ -568,6 +568,63 @@ TEST(CoreOptionsTest, TestLookupCompactMaxIntervalComputedValue) { ASSERT_EQ(13, core_options.GetLookupCompactMaxInterval()); } +TEST(CoreOptionsTest, TestFormatTableOptions) { + // A format table reads these through `CoreOptions` like every other option, so their defaults + // are asserted here rather than a second time in the format table tests. + { + ASSERT_OK_AND_ASSIGN(CoreOptions core_options, CoreOptions::FromMap({})); + ASSERT_FALSE(core_options.FileSuffixIncludeCompression()); + ASSERT_FALSE(core_options.FormatTablePartitionOnlyValueInPath()); + ASSERT_FALSE(core_options.MetastorePartitionedTable()); + } + { + ASSERT_OK_AND_ASSIGN( + CoreOptions core_options, + CoreOptions::FromMap({{Options::FILE_SUFFIX_INCLUDE_COMPRESSION, "true"}, + {Options::FORMAT_TABLE_PARTITION_PATH_ONLY_VALUE, "true"}, + {Options::METASTORE_PARTITIONED_TABLE, "true"}})); + ASSERT_TRUE(core_options.FileSuffixIncludeCompression()); + ASSERT_TRUE(core_options.FormatTablePartitionOnlyValueInPath()); + ASSERT_TRUE(core_options.MetastorePartitionedTable()); + } +} + +TEST(CoreOptionsTest, TestFormatTableFileCompression) { + // One chain, in the order the rest of the paimon ecosystem resolves it: `file.compression`, + // then `format-table.file.compression`, then the bare `compression` key an engine's own + // writer reads, then what the format itself writes. + { + ASSERT_OK_AND_ASSIGN(CoreOptions core_options, + CoreOptions::FromMap({{Options::FILE_COMPRESSION, "lz4"}, + {Options::FORMAT_TABLE_FILE_COMPRESSION, "none"}, + {"compression", "zstd"}})); + ASSERT_EQ(core_options.FormatTableFileCompression(), "lz4"); + } + { + ASSERT_OK_AND_ASSIGN(CoreOptions core_options, + CoreOptions::FromMap({{Options::FORMAT_TABLE_FILE_COMPRESSION, "lz4"}, + {"compression", "zstd"}})); + ASSERT_EQ(core_options.FormatTableFileCompression(), "lz4"); + } + { + ASSERT_OK_AND_ASSIGN(CoreOptions core_options, + CoreOptions::FromMap({{"compression", "lz4"}})); + ASSERT_EQ(core_options.FormatTableFileCompression(), "lz4"); + } + // Left to the format: parquet writes snappy, orc zstd. This differs from + // `GetFileCompression()`, a managed table's `file.compression`, which defaults to zstd. + { + ASSERT_OK_AND_ASSIGN(CoreOptions core_options, CoreOptions::FromMap({})); + ASSERT_EQ(core_options.FormatTableFileCompression(), "snappy"); + ASSERT_EQ(core_options.GetFileCompression(), "zstd"); + } + { + ASSERT_OK_AND_ASSIGN(CoreOptions core_options, + CoreOptions::FromMap({{Options::FILE_FORMAT, "orc"}})); + ASSERT_EQ(core_options.FormatTableFileCompression(), "zstd"); + } +} + TEST(CoreOptionsTest, TestDynamicPartitionOverwriteOption) { { ASSERT_OK_AND_ASSIGN(CoreOptions core_options, CoreOptions::FromMap({})); diff --git a/src/paimon/core/operation/abstract_split_read.cpp b/src/paimon/core/operation/abstract_split_read.cpp index bb82c5d8..86a97419 100644 --- a/src/paimon/core/operation/abstract_split_read.cpp +++ b/src/paimon/core/operation/abstract_split_read.cpp @@ -33,6 +33,7 @@ #include "paimon/common/data/shredding/shredding_file_reader.h" #include "paimon/common/data/variant/variant_shredding_read_plan_factory.h" #include "paimon/common/data/variant/variant_type_utils.h" +#include "paimon/common/reader/data_file_reader_factory.h" #include "paimon/common/reader/delegating_prefetch_reader.h" #include "paimon/common/reader/predicate_batch_reader.h" #include "paimon/common/reader/prefetch_file_batch_reader_impl.h" @@ -124,53 +125,33 @@ Result> AbstractSplitRead::ApplyPredicateFilterIfNe return PredicateBatchReader::Create(std::move(reader), predicate, pool_); } +DataFileReadOptions AbstractSplitRead::DataFileReadOptionsFromContext() const { + DataFileReadOptions read_options; + read_options.cache = options_.GetCache(); + read_options.read_batch_size = options_.GetReadBatchSize(); + read_options.prefetch_enabled = context_->EnablePrefetch(); + read_options.prefetch_max_parallel_num = context_->GetPrefetchMaxParallelNum(); + read_options.prefetch_batch_count = context_->GetPrefetchBatchCount(); + read_options.adaptive_prefetch_strategy = options_.EnableAdaptivePrefetchStrategy(); + read_options.read_ahead_cache_enabled = context_->ReadAheadCacheEnabled(); + read_options.cache_config = context_->GetCacheConfig(); + return read_options; +} + Result> AbstractSplitRead::PrepareReaderBuilder( const std::string& format_identifier, const std::map& extra_format_options) const { - std::map format_options = options_.ToMap(); - // The blob placeholder channels are internal: strip user-supplied blob.internal.* table - // options so only the internal read path can enable them through extra_format_options. - BlobDefs::EraseInternalPlaceholderOptions(&format_options); - for (const auto& [key, value] : extra_format_options) { - format_options[key] = value; - } - PAIMON_ASSIGN_OR_RAISE(std::unique_ptr file_format, - FileFormatFactory::Get(format_identifier, format_options)); - PAIMON_ASSIGN_OR_RAISE(std::unique_ptr reader_builder, - file_format->CreateReaderBuilder(options_.GetReadBatchSize())); - reader_builder->WithMemoryPool(pool_); - reader_builder->WithCache(options_.GetCache()); - // Propagate the framework runtime read state so each format can adapt its own - // behavior (e.g. parquet disabling its pre-buffer when the shared read-ahead cache - // takes over prefetching), instead of mutating format options here. - ReadHints read_hints; - read_hints.prefetch_enabled = context_->EnablePrefetch(); - read_hints.read_ahead_cache_enabled = context_->ReadAheadCacheEnabled(); - reader_builder->WithReadHints(read_hints); - return reader_builder; + return DataFileReaderFactory::CreateReaderBuilder(format_identifier, options_.ToMap(), + extra_format_options, + DataFileReadOptionsFromContext(), pool_); } Result> AbstractSplitRead::CreateFileBatchReader( const std::string& file_format_identifier, const std::string& data_file_path, int64_t data_file_size, const ReaderBuilder* reader_builder) const { - if (context_->EnablePrefetch() && file_format_identifier != "blob" && - file_format_identifier != "avro") { - PAIMON_ASSIGN_OR_RAISE( - std::unique_ptr prefetch_reader, - PrefetchFileBatchReaderImpl::Create( - data_file_path, data_file_size, reader_builder, options_.GetFileSystem(), - context_->GetPrefetchMaxParallelNum(), options_.GetReadBatchSize(), - context_->GetPrefetchBatchCount(), options_.EnableAdaptivePrefetchStrategy(), - executor_, - /*initialize_read_ranges=*/false, context_->ReadAheadCacheEnabled(), - context_->GetCacheConfig(), pool_)); - return std::make_unique(std::move(prefetch_reader)); - } else { - PAIMON_ASSIGN_OR_RAISE( - std::shared_ptr input_stream, - options_.GetFileSystem()->Open(FileStatus(data_file_path, data_file_size))); - return reader_builder->Build(input_stream); - } + return DataFileReaderFactory::Open(file_format_identifier, data_file_path, data_file_size, + reader_builder, DataFileReadOptionsFromContext(), + options_.GetFileSystem(), executor_, pool_); } Result> AbstractSplitRead::CreateFieldMappingReader( diff --git a/src/paimon/core/operation/abstract_split_read.h b/src/paimon/core/operation/abstract_split_read.h index 27349fec..d2ec2eec 100644 --- a/src/paimon/core/operation/abstract_split_read.h +++ b/src/paimon/core/operation/abstract_split_read.h @@ -27,6 +27,7 @@ #include #include "arrow/type_fwd.h" +#include "paimon/common/reader/data_file_reader_factory.h" #include "paimon/core/core_options.h" #include "paimon/core/deletionvectors/deletion_vector.h" #include "paimon/core/io/field_mapping_reader.h" @@ -101,6 +102,10 @@ class AbstractSplitRead : public SplitRead { const std::optional>& write_cols); private: + /// What a data file is opened with, gathered from the context and the table options once so + /// that the reader builder and the file reader cannot be built from different answers. + DataFileReadOptions DataFileReadOptionsFromContext() const; + Result> PrepareReaderBuilder( const std::string& format_identifier, const std::map& extra_format_options) const; diff --git a/src/paimon/core/operation/file_store_commit.cpp b/src/paimon/core/operation/file_store_commit.cpp index ad0942ad..a80a13e9 100644 --- a/src/paimon/core/operation/file_store_commit.cpp +++ b/src/paimon/core/operation/file_store_commit.cpp @@ -20,6 +20,7 @@ #include "paimon/file_store_commit.h" #include +#include #include #include "paimon/commit_context.h" @@ -37,12 +38,16 @@ #include "paimon/core/operation/key_value_file_store_scan.h" #include "paimon/core/schema/schema_manager.h" #include "paimon/core/schema/table_schema.h" +#include "paimon/core/table/format/format_table_file_store_commit.h" +#include "paimon/core/table/format/format_table_loader.h" +#include "paimon/core/utils/branch_manager.h" #include "paimon/core/utils/field_mapping.h" #include "paimon/core/utils/file_store_path_factory.h" #include "paimon/core/utils/snapshot_manager.h" #include "paimon/format/file_format.h" #include "paimon/fs/file_system.h" #include "paimon/result.h" +#include "paimon/table/format/format_table.h" namespace arrow { class Schema; @@ -109,13 +114,44 @@ Result> FileStoreCommit::Create( PAIMON_ASSIGN_OR_RAISE(CoreOptions tmp_options, CoreOptions::FromMap(ctx->GetOptions(), ctx->GetSpecificFileSystem())); const std::string& root_path = ctx->GetRootPath(); + // A format table commits by renaming files into place, so it never reaches the snapshot path + // below. The managed path here reads the main branch, so this reads the same one: the two + // must not dispatch on different schemas. auto schema_manager = std::make_shared(tmp_options.GetFileSystem(), root_path); - PAIMON_ASSIGN_OR_RAISE(std::optional> table_schema, - schema_manager->Latest()); - if (table_schema == std::nullopt) { + std::shared_ptr latest_schema; + PAIMON_ASSIGN_OR_RAISE( + std::shared_ptr format_table, + FormatTableLoader::TryLoad(tmp_options.GetFileSystem(), root_path, + BranchManager::DEFAULT_MAIN_BRANCH, ctx->GetOptions(), + /*specific_table_schema=*/std::nullopt, schema_manager.get(), + &latest_schema)); + if (format_table != nullptr) { + // Anything the context carries that a format table cannot honour is refused rather than + // silently dropped. Each of the three is refused only when set away from its default. + if (!ctx->IgnoreEmptyCommit()) { + return Status::NotImplemented( + "a format table cannot record an empty commit: keeping one means writing a " + "snapshot that adds no files, and there are no snapshots here"); + } + if (ctx->UseRESTCatalogCommit()) { + return Status::NotImplemented( + "a format table commits by renaming files into place, not by sending a snapshot " + "to a rest catalog"); + } + if (ctx->AppendCommitCheckConflict()) { + return Status::NotImplemented( + "a format table has no manifests to check a concurrent commit against"); + } + PAIMON_ASSIGN_OR_RAISE(std::unique_ptr format_commit, + FormatTableFileStoreCommit::Create(format_table)); + return std::unique_ptr(std::move(format_commit)); + } + // The schema the dispatch above already read through `schema_manager`, rather than a second + // read of the same file. + if (latest_schema == nullptr) { return Status::Invalid("not found latest schema"); } - const auto& schema = table_schema.value(); + const std::shared_ptr& schema = latest_schema; auto opts = schema->Options(); for (const auto& [key, value] : ctx->GetOptions()) { opts[key] = value; @@ -134,11 +170,11 @@ Result> FileStoreCommit::Create( return Status::NotImplemented( "commit operation does not support object store file system for now"); } - PAIMON_ASSIGN_OR_RAISE( - std::unique_ptr partition_computer, - BinaryRowPartitionComputer::Create( - table_schema.value()->PartitionKeys(), arrow_schema, options.GetPartitionDefaultName(), - options.LegacyPartitionNameEnabled(), ctx->GetMemoryPool())); + PAIMON_ASSIGN_OR_RAISE(std::unique_ptr partition_computer, + BinaryRowPartitionComputer::Create(schema->PartitionKeys(), arrow_schema, + options.GetPartitionDefaultName(), + options.LegacyPartitionNameEnabled(), + ctx->GetMemoryPool())); PAIMON_ASSIGN_OR_RAISE(std::vector external_paths, options.CreateExternalPaths()); PAIMON_ASSIGN_OR_RAISE(std::optional global_index_external_path, options.CreateGlobalIndexExternalPath()); @@ -146,10 +182,10 @@ Result> FileStoreCommit::Create( PAIMON_ASSIGN_OR_RAISE( std::shared_ptr path_factory, FileStorePathFactory::Create( - root_path, arrow_schema, table_schema.value()->PartitionKeys(), - options.GetPartitionDefaultName(), options.GetFileFormat()->Identifier(), - options.DataFilePrefix(), options.LegacyPartitionNameEnabled(), external_paths, - global_index_external_path, options.IndexFileInDataFileDir(), ctx->GetMemoryPool())); + root_path, arrow_schema, schema->PartitionKeys(), options.GetPartitionDefaultName(), + options.GetFileFormat()->Identifier(), options.DataFilePrefix(), + options.LegacyPartitionNameEnabled(), external_paths, global_index_external_path, + options.IndexFileInDataFileDir(), ctx->GetMemoryPool())); auto snapshot_manager = std::make_shared(options.GetFileSystem(), root_path); PAIMON_ASSIGN_OR_RAISE( @@ -158,9 +194,8 @@ Result> FileStoreCommit::Create( options.GetManifestCompression(), path_factory, options.GetCache(), ctx->GetMemoryPool())); - PAIMON_ASSIGN_OR_RAISE( - std::shared_ptr partition_schema, - FieldMapping::GetPartitionSchema(arrow_schema, table_schema.value()->PartitionKeys())); + PAIMON_ASSIGN_OR_RAISE(std::shared_ptr partition_schema, + FieldMapping::GetPartitionSchema(arrow_schema, schema->PartitionKeys())); PAIMON_ASSIGN_OR_RAISE( std::shared_ptr manifest_file, ManifestFile::Create(options.GetFileSystem(), options.GetManifestFormat(), @@ -178,22 +213,22 @@ Result> FileStoreCommit::Create( options.GetExpireConfig(), options.RealtimeEnabled(), ctx->GetExecutor()); CommitScanner::ScanSupplier scan_supplier; - if (table_schema.value()->PrimaryKeys().empty()) { + if (schema->PrimaryKeys().empty()) { scan_supplier = CreateAppendScanSupplier(snapshot_manager, schema_manager, manifest_list, - manifest_file, table_schema.value(), arrow_schema, - options, ctx->GetExecutor(), ctx->GetMemoryPool()); + manifest_file, schema, arrow_schema, options, + ctx->GetExecutor(), ctx->GetMemoryPool()); } else { scan_supplier = CreatePkScanSupplier(snapshot_manager, schema_manager, manifest_list, - manifest_file, table_schema.value(), arrow_schema, - options, ctx->GetExecutor(), ctx->GetMemoryPool()); + manifest_file, schema, arrow_schema, options, + ctx->GetExecutor(), ctx->GetMemoryPool()); } return std::make_unique( ctx->GetMemoryPool(), ctx->GetExecutor(), arrow_schema, root_path, ctx->GetCommitUser(), options, path_factory, std::move(partition_computer), snapshot_manager, ctx->IgnoreEmptyCommit(), ctx->UseRESTCatalogCommit(), ctx->AppendCommitCheckConflict(), - table_schema.value(), manifest_file, manifest_list, index_manifest_file, expire_snapshots, - schema_manager, std::move(scan_supplier)); + schema, manifest_file, manifest_list, index_manifest_file, expire_snapshots, schema_manager, + std::move(scan_supplier)); } } // namespace paimon diff --git a/src/paimon/core/operation/file_store_write.cpp b/src/paimon/core/operation/file_store_write.cpp index 6807ae35..2cee533a 100644 --- a/src/paimon/core/operation/file_store_write.cpp +++ b/src/paimon/core/operation/file_store_write.cpp @@ -40,12 +40,15 @@ #include "paimon/core/schema/schema_manager.h" #include "paimon/core/schema/table_schema.h" #include "paimon/core/table/bucket_mode.h" +#include "paimon/core/table/format/format_table_file_store_write.h" +#include "paimon/core/table/format/format_table_loader.h" #include "paimon/core/utils/field_mapping.h" #include "paimon/core/utils/file_store_path_factory.h" #include "paimon/core/utils/primary_key_table_utils.h" #include "paimon/core/utils/snapshot_manager.h" #include "paimon/format/file_format.h" #include "paimon/result.h" +#include "paimon/table/format/format_table.h" #include "paimon/write_context.h" namespace arrow { @@ -80,14 +83,50 @@ Result> FileStoreWrite::Create(std::unique_ptrGetOptions(), ctx->GetSpecificFileSystem(), ctx->GetFileSystemSchemeToIdentifierMap())); std::string branch = ctx->GetBranch(); + // A format table writes plain data files into a directory, so it never reaches the manifest + // path below. One `FileStoreWrite` interface serves both, as Java Paimon serves both through + // one `BatchWriteBuilder`. auto schema_manager = std::make_shared(tmp_options.GetFileSystem(), ctx->GetRootPath(), branch); - PAIMON_ASSIGN_OR_RAISE(std::optional> table_schema, - schema_manager->Latest()); - if (table_schema == std::nullopt) { + std::shared_ptr latest_schema; + PAIMON_ASSIGN_OR_RAISE( + std::shared_ptr format_table, + FormatTableLoader::TryLoad(tmp_options.GetFileSystem(), ctx->GetRootPath(), branch, + ctx->GetOptions(), /*specific_table_schema=*/std::nullopt, + schema_manager.get(), &latest_schema)); + if (format_table != nullptr) { + // Anything the context carries that a format table cannot honour is refused rather than + // silently dropped. + if (ctx->IsStreamingMode()) { + return Status::NotImplemented( + "a format table has no snapshots, so there is nothing a streaming write could " + "commit against"); + } + if (ctx->GetRealtimeContext() != nullptr) { + return Status::NotImplemented("a format table has no real-time store to write into"); + } + if (!ctx->GetWriteSchema().empty()) { + return Status::NotImplemented( + "a format table write takes the table's own columns; a write schema naming a " + "subset of them is not supported yet"); + } + // A write id prefixes a postpone-bucket writer's files so that one compaction reader can + // put them back in order. A format table has no buckets, so it would identify nothing. + if (ctx->GetWriteId().has_value()) { + return Status::NotImplemented( + "a format table has no buckets, so a write id would name nothing"); + } + PAIMON_ASSIGN_OR_RAISE( + std::unique_ptr format_write, + FormatTableFileStoreWrite::Create(format_table, ctx->GetMemoryPool())); + return std::unique_ptr(std::move(format_write)); + } + // The schema the dispatch above already read through `schema_manager`, rather than a second + // read of the same file. + if (latest_schema == nullptr) { return Status::Invalid(fmt::format("cannot found latest schema in branch {}", branch)); } - const auto& schema = table_schema.value(); + const std::shared_ptr& schema = latest_schema; auto opts = schema->Options(); for (const auto& [key, value] : ctx->GetOptions()) { opts[key] = value; diff --git a/src/paimon/core/options/table_type.h b/src/paimon/core/options/table_type.h new file mode 100644 index 00000000..c4233574 --- /dev/null +++ b/src/paimon/core/options/table_type.h @@ -0,0 +1,98 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +#pragma once + +#include +#include + +#include "fmt/format.h" +#include "paimon/common/utils/string_utils.h" +#include "paimon/defs.h" +#include "paimon/result.h" +#include "paimon/status.h" + +namespace paimon { + +/// Type of a table, carried by the `type` table option. +/// +/// Every type paimon defines is named here, even the ones this library cannot open: telling such +/// a table from a managed one means knowing its name. +enum class TableType { + /// A managed paimon table with snapshots and manifests. + TABLE = 1, + /// A directory of data files laid out like a standard Hive table, without paimon metadata. + FORMAT_TABLE = 2, + /// A managed paimon table that also carries the SQL it materializes. + MATERIALIZED_TABLE = 3, + /// A managed paimon table over the objects of a location. + OBJECT_TABLE = 4, + /// A lance table, see 'https://lancedb.github.io/lance/'. + LANCE_TABLE = 5, + /// An iceberg table, see 'https://iceberg.apache.org/'. + ICEBERG_TABLE = 6, +}; + +/// Identifiers of `TableType` as they appear in the `type` table option. +struct TableTypeDefine { + static constexpr char kTable[] = "table"; + static constexpr char kFormatTable[] = "format-table"; + static constexpr char kMaterializedTable[] = "materialized-table"; + static constexpr char kObjectTable[] = "object-table"; + static constexpr char kLanceTable[] = "lance-table"; + static constexpr char kIcebergTable[] = "iceberg-table"; + + /// Reads the table type out of a table option map. + /// + /// An absent `type` is `TableType::TABLE`, and the value is matched without regard to case. + /// A value that names no table type is rejected rather than read as a managed table. + static Result FromOptions(const std::map& options) { + auto iter = options.find(Options::TYPE); + if (iter == options.end()) { + return TableType::TABLE; + } + std::string value = StringUtils::ToLowerCase(iter->second); + if (value == kTable) { + return TableType::TABLE; + } + if (value == kFormatTable) { + return TableType::FORMAT_TABLE; + } + if (value == kMaterializedTable) { + return TableType::MATERIALIZED_TABLE; + } + if (value == kObjectTable) { + return TableType::OBJECT_TABLE; + } + if (value == kLanceTable) { + return TableType::LANCE_TABLE; + } + if (value == kIcebergTable) { + return TableType::ICEBERG_TABLE; + } + return Status::Invalid(fmt::format("unknown table type: {}", iter->second)); + } + + static Result IsFormatTable(const std::map& options) { + PAIMON_ASSIGN_OR_RAISE(TableType table_type, FromOptions(options)); + return table_type == TableType::FORMAT_TABLE; + } +}; + +} // namespace paimon diff --git a/src/paimon/core/schema/schema_manager.cpp b/src/paimon/core/schema/schema_manager.cpp index 2425cc4a..8eb53195 100644 --- a/src/paimon/core/schema/schema_manager.cpp +++ b/src/paimon/core/schema/schema_manager.cpp @@ -22,10 +22,12 @@ #include #include +#include "fmt/format.h" #include "paimon/common/utils/path_util.h" #include "paimon/core/schema/schema_validation.h" #include "paimon/core/utils/branch_manager.h" #include "paimon/core/utils/file_utils.h" +#include "paimon/defs.h" #include "paimon/fs/file_system.h" #include "paimon/status.h" @@ -110,7 +112,8 @@ Result> SchemaManager::CreateTable( PAIMON_ASSIGN_OR_RAISE( std::unique_ptr table_schema, TableSchema::Create(/*schema_id=*/0, schema, partition_keys, primary_keys, options)); - PAIMON_RETURN_NOT_OK(SchemaValidation::ValidateTableSchema(*table_schema)); + // Shared with every other catalog, so a schema this one persists is one they all open. + PAIMON_RETURN_NOT_OK(SchemaValidation::ValidateNewTableSchema(*table_schema)); std::string schema_path = ToSchemaPath(0); PAIMON_ASSIGN_OR_RAISE(std::string content, table_schema->ToJsonString()); auto status = file_system_->AtomicStore(schema_path, content); diff --git a/src/paimon/core/schema/schema_validation.cpp b/src/paimon/core/schema/schema_validation.cpp index 7342826d..855da7be 100644 --- a/src/paimon/core/schema/schema_validation.cpp +++ b/src/paimon/core/schema/schema_validation.cpp @@ -23,6 +23,7 @@ #include #include #include +#include #include #include #include @@ -30,6 +31,8 @@ #include #include +#include "arrow/c/bridge.h" +#include "arrow/c/helpers.h" #include "arrow/type.h" #include "fmt/format.h" #include "fmt/ranges.h" @@ -38,21 +41,27 @@ #include "paimon/common/data/variant/variant_type_utils.h" #include "paimon/common/table/special_fields.h" #include "paimon/common/types/data_field.h" +#include "paimon/common/utils/arrow/status_utils.h" #include "paimon/common/utils/arrow/vector_utils.h" +#include "paimon/common/utils/binary_row_partition_computer.h" #include "paimon/common/utils/checked_cast.h" #include "paimon/common/utils/object_utils.h" +#include "paimon/common/utils/options_utils.h" #include "paimon/common/utils/preconditions.h" +#include "paimon/common/utils/scope_guard.h" #include "paimon/common/utils/string_utils.h" #include "paimon/core/core_options.h" #include "paimon/core/options/changelog_producer.h" #include "paimon/core/options/expire_config.h" #include "paimon/core/options/map_storage_layout.h" #include "paimon/core/options/merge_engine.h" +#include "paimon/core/options/table_type.h" #include "paimon/core/schema/arrow_schema_validator.h" #include "paimon/core/schema/table_schema.h" #include "paimon/core/table/bucket_mode.h" #include "paimon/defs.h" #include "paimon/result.h" +#include "paimon/table/format/format_table.h" namespace paimon { namespace { @@ -134,27 +143,146 @@ bool SchemaValidation::IsComplexType(const std::shared_ptr& field) BlobUtils::IsBlobField(field)); } -Status SchemaValidation::ValidateTableSchema(const TableSchema& schema) { - const auto& field_names = schema.FieldNames(); - PAIMON_RETURN_NOT_OK(ValidateNoDuplicateField(schema.BucketKeys(), "bucket key")); - PAIMON_RETURN_NOT_OK(ValidateNoDuplicateField(schema.PrimaryKeys(), "primary key")); - PAIMON_RETURN_NOT_OK(ValidateNoDuplicateField(schema.PartitionKeys(), "partition key")); - PAIMON_RETURN_NOT_OK( - Preconditions::CheckState(ObjectUtils::ContainsAll(field_names, schema.PartitionKeys()), - "Table column {} should include all partition fields {}", - field_names, schema.PartitionKeys())); - PAIMON_RETURN_NOT_OK( - Preconditions::CheckState(ObjectUtils::ContainsAll(field_names, schema.PrimaryKeys()), - "Table column {} should include all primary key constraint {}", - field_names, schema.PrimaryKeys())); - - PAIMON_RETURN_NOT_OK( - ValidateOnlyContainPrimitiveType(schema.Fields(), schema.PrimaryKeys(), "primary key")); - PAIMON_RETURN_NOT_OK( - ValidateOnlyContainPrimitiveType(schema.Fields(), schema.PartitionKeys(), "partition")); +Status SchemaValidation::ValidateGenericSchema(const std::vector& fields, + const std::vector& bucket_keys, + const std::vector& primary_keys, + const std::vector& partition_keys) { + std::vector field_names; + field_names.reserve(fields.size()); + for (const DataField& field : fields) { + field_names.push_back(field.Name()); + } + PAIMON_RETURN_NOT_OK(ValidateNoDuplicateField(bucket_keys, "bucket key")); + PAIMON_RETURN_NOT_OK(ValidateNoDuplicateField(primary_keys, "primary key")); + PAIMON_RETURN_NOT_OK(ValidateNoDuplicateField(partition_keys, "partition key")); + PAIMON_RETURN_NOT_OK(Preconditions::CheckState( + ObjectUtils::ContainsAll(field_names, partition_keys), + "Table column {} should include all partition fields {}", field_names, partition_keys)); + PAIMON_RETURN_NOT_OK(Preconditions::CheckState( + ObjectUtils::ContainsAll(field_names, primary_keys), + "Table column {} should include all primary key constraint {}", field_names, primary_keys)); + for (const auto& field_name : field_names) { + if (SpecialFields::IsSystemField(field_name)) { + return Status::Invalid( + fmt::format("field name '{}' in schema cannot be special field.", field_name)); + } + } + PAIMON_RETURN_NOT_OK(ValidateOnlyContainPrimitiveType(fields, primary_keys, "primary key")); + PAIMON_RETURN_NOT_OK(ValidateOnlyContainPrimitiveType(fields, partition_keys, "partition")); // TODO(lisizhuo.lsz): C++ Paimon do not support timestamp & decimal & float & double type in // partition keys for now. - PAIMON_RETURN_NOT_OK(ValidateNotContainSpecificType(schema.Fields(), schema.PartitionKeys())); + PAIMON_RETURN_NOT_OK(ValidateNotContainSpecificType(fields, partition_keys)); + return Status::OK(); +} + +Status SchemaValidation::ValidateNewTableSchema(const TableSchema& schema) { + const std::map& options = schema.Options(); + PAIMON_ASSIGN_OR_RAISE(TableType table_type, TableTypeDefine::FromOptions(options)); + if (table_type != TableType::TABLE && table_type != TableType::MATERIALIZED_TABLE && + table_type != TableType::FORMAT_TABLE) { + // Quoted back rather than re-rendered from `table_type`, so the message says what was + // actually asked for. + auto type_iter = options.find(Options::TYPE); + return Status::NotImplemented(fmt::format( + "Cannot create a table whose '{}' is '{}': paimon-cpp does not implement this table " + "type.", + Options::TYPE, type_iter == options.end() ? std::string() : type_iter->second)); + } + if (table_type == TableType::FORMAT_TABLE) { + PAIMON_RETURN_NOT_OK(ValidateGenericTableSchema(schema)); + // At creation the schema's own options are the only ones there are, and `file-system` is + // resolved from them like every other option. + return ValidateFormatTableSchema(schema, schema.Options(), /*file_system=*/nullptr); + } + return ValidateTableSchema(schema); +} + +Status SchemaValidation::ValidateGenericTableSchema(const TableSchema& schema) { + return ValidateGenericSchema(schema.Fields(), schema.BucketKeys(), schema.PrimaryKeys(), + schema.PartitionKeys()); +} + +Status SchemaValidation::ValidateGenericDataSchema(const DataSchema& schema) { + // A `DataSchema` names its field types through its arrow schema rather than through + // `DataField`s, so they are converted back and the same rules run on them. + PAIMON_ASSIGN_OR_RAISE(std::unique_ptr<::ArrowSchema> c_schema, schema.GetArrowSchema()); + ScopeGuard schema_guard([&c_schema]() { ArrowSchemaRelease(c_schema.get()); }); + PAIMON_ASSIGN_OR_RAISE_FROM_ARROW(std::shared_ptr arrow_schema, + arrow::ImportSchema(c_schema.get())); + PAIMON_ASSIGN_OR_RAISE(std::vector fields, + DataField::ConvertArrowSchemaToDataFields(arrow_schema)); + return ValidateGenericSchema(fields, schema.BucketKeys(), schema.PrimaryKeys(), + schema.PartitionKeys()); +} + +Status SchemaValidation::ValidateFormatTableSchema( + const DataSchema& schema, const std::map& effective_options, + const std::shared_ptr& file_system) { + // Runs both when the table is created and when it is opened: creation alone would let a + // schema written elsewhere through, and opening alone would persist a table nothing can load. + if (!schema.PrimaryKeys().empty()) { + return Status::Invalid( + "Cannot define primary keys for a format table: a directory of data files records no " + "row identity to merge on."); + } + + PAIMON_ASSIGN_OR_RAISE(std::string file_format, + OptionsUtils::GetValueFromMap( + effective_options, Options::FILE_FORMAT, "parquet")); + // Before `CoreOptions`, which resolves `file.format` through the format factories and would + // fail with a missing-factory error where this names the format. The parsed value is not + // kept; this only has to fail for a format nothing here can read. + PAIMON_RETURN_NOT_OK(FormatTable::ParseFormat(file_format)); + + // The remaining options are read through `CoreOptions`, so a default only ever changes in one + // place. `target-file-row-num` is validated there, so it needs no check of its own here. + PAIMON_ASSIGN_OR_RAISE(CoreOptions core_options, + CoreOptions::FromMap(effective_options, file_system)); + + if (core_options.FormatTablePartitionOnlyValueInPath() && schema.PartitionKeys().empty()) { + return Status::Invalid( + "Cannot set 'format-table.partition-path-only-value' on a table with no partition " + "keys: the layout names a directory by its partition value alone."); + } + + // A table of nothing but partition columns leaves a write nothing to write and a read no + // column to count rows by. + if (!schema.PartitionKeys().empty() && + schema.PartitionKeys().size() == schema.FieldNames().size()) { + return Status::Invalid( + "A format table cannot be partitioned by every one of its columns: the data files " + "would hold nothing, since partition values live in the directory names."); + } + + // A partition value makes the round trip through its column type on the way to a directory + // name and back, so a type that cannot make it leaves a table nothing can read or write. + // Checked by building the computer that does the round trip rather than by listing the types + // it accepts, which would be a second list to keep in step with the first. + if (!schema.PartitionKeys().empty()) { + PAIMON_ASSIGN_OR_RAISE(std::unique_ptr<::ArrowSchema> c_schema, schema.GetArrowSchema()); + PAIMON_ASSIGN_OR_RAISE_FROM_ARROW(std::shared_ptr arrow_schema, + arrow::ImportSchema(c_schema.get())); + Result> partition_computer = + BinaryRowPartitionComputer::Create( + schema.PartitionKeys(), arrow_schema, core_options.GetPartitionDefaultName(), + core_options.LegacyPartitionNameEnabled(), GetDefaultPool()); + if (!partition_computer.ok()) { + return Status(partition_computer.status().code(), + fmt::format("a format table cannot be partitioned by these columns: {}", + partition_computer.status().message())); + } + } + + if (core_options.MetastorePartitionedTable()) { + return Status::NotImplemented( + "'metastore.partitioned-table' is not supported by paimon-cpp yet: its partitions " + "would come from the catalog rather than from the directories a scan here reads."); + } + return Status::OK(); +} + +Status SchemaValidation::ValidateTableSchema(const TableSchema& schema) { + PAIMON_RETURN_NOT_OK(ValidateGenericTableSchema(schema)); PAIMON_ASSIGN_OR_RAISE(CoreOptions options, CoreOptions::FromMap(schema.Options())); PAIMON_RETURN_NOT_OK(ValidateBucket(schema, options)); @@ -182,12 +310,6 @@ Status SchemaValidation::ValidateTableSchema(const TableSchema& schema) { // TODO(yonghao.fyh): check changelog num retain // TODO(yonghao.fyh): support file format validate data fields - for (const auto& field_name : field_names) { - if (SpecialFields::IsSystemField(field_name)) { - return Status::Invalid( - fmt::format("field name '{}' in schema cannot be special field.", field_name)); - } - } // TODO(yonghao.fyh): check streaming read overwrite // TODO(yonghao.fyh): check 'partition.expiration-time' // TODO(yonghao.fyh): check 'rowkind.field' diff --git a/src/paimon/core/schema/schema_validation.h b/src/paimon/core/schema/schema_validation.h index abf4d5b0..dfa58a76 100644 --- a/src/paimon/core/schema/schema_validation.h +++ b/src/paimon/core/schema/schema_validation.h @@ -20,6 +20,7 @@ #pragma once #include +#include #include #include #include @@ -36,6 +37,8 @@ class Field; namespace paimon { class CoreOptions; class DataField; +class DataSchema; +class FileSystem; class TableSchema; /// Validation utils for `TableSchema`. @@ -46,11 +49,52 @@ class SchemaValidation { static Status ValidateTableSchema(const TableSchema& schema); + /// Validates a schema a table is about to be created from, whichever catalog is creating it. + /// + /// It picks the rules by the schema's `type` option: a format table owns no bucket, manifest + /// or snapshot machinery, so only the structural invariants apply to it. A type this library + /// cannot load at all is refused rather than persisted, since the table would look created + /// and fail only when someone tried to open it. + static Status ValidateNewTableSchema(const TableSchema& schema); + + /// Validates the structural invariants every table shares, whatever its type: partition and + /// primary key fields exist and are free of duplicates, key fields are primitive, and no field + /// takes a reserved name. + static Status ValidateGenericTableSchema(const TableSchema& schema); + + /// The same rules as `ValidateGenericTableSchema()`, for a schema this library did not + /// create. A `DataSchema` names its field types through its arrow schema rather than through + /// `DataField`s, which is the only reason this is a second entry point. + static Status ValidateGenericDataSchema(const DataSchema& schema); + + /// Validates what a format table additionally requires, on top of + /// `ValidateGenericTableSchema()`. It takes a `DataSchema` so that it can run both at creation + /// and at `FormatTable::Create()`, where the schema may never have passed through creation + /// here at all. + /// + /// @param effective_options The options the table will actually run with: the schema's own at + /// creation, and those with anything given at the call merged on top at + /// `FormatTable::Create()`. Passing the schema's alone would let an option given at the + /// call reach a table that refuses it in its schema. + /// @param file_system The file system the caller already resolved, or null when it named one + /// through the options. It only spares reading the options from resolving `file-system` + /// a second time, which would fail for a caller that handed its own over instead of + /// naming one. + static Status ValidateFormatTableSchema( + const DataSchema& schema, const std::map& effective_options, + const std::shared_ptr& file_system); + static bool IsPostponeBucketTable(const TableSchema& schema, int32_t bucket); private: static Status ValidateNoDuplicateField(const std::vector& field_names, const std::string& error_message_intro); + /// The rules `ValidateGenericTableSchema()` and `ValidateGenericDataSchema()` share, on the + /// fields both can hand over. + static Status ValidateGenericSchema(const std::vector& fields, + const std::vector& bucket_keys, + const std::vector& primary_keys, + const std::vector& partition_keys); static Status ValidateOnlyContainPrimitiveType(const std::vector& fields, const std::vector& field_names, const std::string& error_message_intro); diff --git a/src/paimon/core/table/format/format_commit_message.h b/src/paimon/core/table/format/format_commit_message.h new file mode 100644 index 00000000..90ee94cd --- /dev/null +++ b/src/paimon/core/table/format/format_commit_message.h @@ -0,0 +1,65 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#pragma once + +#include +#include +#include + +#include "paimon/commit_message.h" + +namespace paimon { + +/// One file a `FormatTableWrite` has written but not yet published. +/// +/// The file is complete on disk under `temp_file_path`, which a scan skips; committing renames it +/// to `file_path`, which is what makes it part of the table. +/// +/// It is a `CommitMessage` so that a format table can be written and committed through +/// `FileStoreWrite` and `FileStoreCommit` like any other table, as Java Paimon's +/// `TwoPhaseCommitMessage` is. It names a staged path rather than files to record in a manifest, +/// so `CommitMessage::Serialize()` refuses it: there is no cross-runtime encoding for one, and a +/// write and its commit belong to the same process. +struct FormatCommitMessage : public CommitMessage { + FormatCommitMessage(const std::string& _temp_file_path, const std::string& _file_path, + const std::map& _partition, int64_t _record_count, + int64_t _file_size) + : temp_file_path(_temp_file_path), + file_path(_file_path), + partition(_partition), + record_count(_record_count), + file_size(_file_size) {} + + ~FormatCommitMessage() override = default; + + std::string ToString() const; + + /// Path the data was written to: a hidden file a scan skips. + std::string temp_file_path; + /// Path the file is renamed to when the write is committed. + std::string file_path; + /// Partition the file belongs to, empty when the table is not partitioned. + std::map partition; + /// Rows written to the file. + int64_t record_count; + /// Size of the written file in bytes. + int64_t file_size; +}; + +} // namespace paimon diff --git a/src/paimon/core/table/format/format_data_split.h b/src/paimon/core/table/format/format_data_split.h new file mode 100644 index 00000000..d9c1c84c --- /dev/null +++ b/src/paimon/core/table/format/format_data_split.h @@ -0,0 +1,88 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#pragma once + +#include +#include +#include +#include +#include + +#include "paimon/table/source/split.h" + +namespace paimon { + +/// A split of a format table: the data files of one partition directory, or of the table +/// directory itself when the table is not partitioned. +/// +/// A file is never divided between splits, because parquet and orc record their own row group and +/// stripe boundaries and a reader handed a byte range would have to rediscover them. The partition +/// is carried on the split rather than read from the files: a Hive-style layout keeps partition +/// values in the directory names. +/// +/// In-memory only: `Split::Serialize()` refuses it, since a format table's plan has no +/// cross-runtime encoding. Plan and read within one process. +struct FormatDataSplit : public Split { + /// One data file of the split. + struct FileMeta { + FileMeta(const std::string& _file_path, int64_t _file_size) + : file_path(_file_path), file_size(_file_size) {} + + bool operator==(const FileMeta& other) const { + return file_path == other.file_path && file_size == other.file_size; + } + + /// Absolute path of the data file. + std::string file_path; + /// Size of the data file in bytes, as reported by the listing that found it. + int64_t file_size; + }; + + FormatDataSplit(const std::vector& files, + const std::map& partition) + : files(files), partition(partition) {} + + ~FormatDataSplit() override = default; + + /// Total size of the split in bytes, i.e. the sum of every file's size. + /// + /// The sizes are whatever the split was given, so a total that would not fit an int64 + /// saturates instead of wrapping into a negative answer. + int64_t TotalSize() const { + int64_t total = 0; + for (const FileMeta& file : files) { + if (file.file_size > 0 && + total > std::numeric_limits::max() - file.file_size) { + return std::numeric_limits::max(); + } + total += file.file_size; + } + return total; + } + + /// Data files of this split, read in this order. + std::vector files; + + /// Partition values shared by every row of this split, keyed by partition field name. Empty + /// when the table is not partitioned. A value equal to the table's `partition.default-name` + /// stands for a null partition value. + std::map partition; +}; + +} // namespace paimon diff --git a/src/paimon/core/table/format/format_file_listing.cpp b/src/paimon/core/table/format/format_file_listing.cpp new file mode 100644 index 00000000..3ff6e2ab --- /dev/null +++ b/src/paimon/core/table/format/format_file_listing.cpp @@ -0,0 +1,91 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include "paimon/core/table/format/format_file_listing.h" + +#include + +#include "fmt/format.h" +#include "paimon/common/utils/path_util.h" +#include "paimon/core/utils/partition_path_utils.h" +#include "paimon/fs/file_system.h" + +namespace paimon { + +bool FormatFileListing::IsReservedDirectory(const std::string& name) { + return name == "schema" || name == "branch"; +} + +Status FormatFileListing::ListDataFiles(const std::shared_ptr& file_system, + const std::string& root, + const FormatDataFileListingOptions& options, + std::vector* files) { + // Checked explicitly: the file systems here report a missing directory as an empty listing + // rather than as an error. + PAIMON_ASSIGN_OR_RAISE(FileStatus root_status, file_system->GetFileStatus(root)); + if (!root_status.IsDir()) { + return Status::Invalid( + fmt::format("{} is not a directory, so it holds no table data", root)); + } + + std::vector level = {root}; + // Depth of the directories in `level`, counted from the listed root. + int32_t depth = 0; + while (!level.empty()) { + // The default partition name holds table content only at a partition level; anywhere else + // a hidden name is a staging tree. + const bool children_are_partitions = options.partition_levels >= depth + 1; + const bool exempt_default_part_name = children_are_partitions && + options.only_value_in_path && + !options.default_part_name.empty(); + std::vector next; + for (const std::string& directory : level) { + std::vector children; + Status status = file_system->ListFileStatus(directory, &children); + if (status.IsNotExist()) { + // Gone since its parent listed it; the rest still stands. Not the root, which + // was checked above. + continue; + } + PAIMON_RETURN_NOT_OK(status); + for (const FileStatus& child : children) { + const std::string name = PathUtil::GetName(child.GetPath()); + const bool hidden = PartitionPathUtils::IsHiddenName(name); + if (child.IsDir()) { + const bool is_default_part_dir = + exempt_default_part_name && name == options.default_part_name; + if (hidden && !is_default_part_dir) { + continue; + } + if (depth == 0 && options.skip_reserved_directories && + IsReservedDirectory(name)) { + continue; + } + next.push_back(child.GetPath()); + } else if (!hidden) { + files->emplace_back(child.GetPath(), child.GetLen()); + } + } + } + level = std::move(next); + depth++; + } + return Status::OK(); +} + +} // namespace paimon diff --git a/src/paimon/core/table/format/format_file_listing.h b/src/paimon/core/table/format/format_file_listing.h new file mode 100644 index 00000000..47ab52df --- /dev/null +++ b/src/paimon/core/table/format/format_file_listing.h @@ -0,0 +1,72 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#pragma once + +#include +#include +#include +#include + +#include "paimon/core/table/format/format_data_split.h" +#include "paimon/status.h" + +namespace paimon { + +class FileSystem; + +/// What the listing has to know about the directory tree below its root. +struct FormatDataFileListingOptions { + /// Levels below the root holding partition directories rather than table content. Zero when + /// the root is already a complete partition. + int32_t partition_levels = 0; + /// Whether a partition directory is named by its value alone instead of `key=value`. + bool only_value_in_path = false; + /// The name standing for a null partition value: in the value-only layout, the one hidden + /// name that holds table content. + std::string default_part_name; + /// Whether `schema` and `branch` below the root are this table's metadata. False when the + /// schema lives in a metastore, where either name is data. + bool skip_reserved_directories = false; +}; + +/// Finds the data files of a format table in the directory tree it is laid out in. +class FormatFileListing { + public: + FormatFileListing() = delete; + ~FormatFileListing() = delete; + + /// Whether `name` is a directory this library keeps under a format table's location as + /// metadata. A table whose schema lives in a catalog has none. + static bool IsReservedDirectory(const std::string& name); + + /// Collects every committed data file under `root`, at any depth. + /// + /// A hidden `_` / `.` name is skipped and never descended into: an uncommitted job stages + /// output there under ordinary data file names, so only the directory above tells them apart. + /// The one exception is `default_part_name`, and only at a partition level. + /// + /// A root that does not exist is an error: the location is wrong or the data is gone, which is + /// not the same as a table with no rows. A directory that disappears mid-listing is skipped. + static Status ListDataFiles(const std::shared_ptr& file_system, + const std::string& root, + const FormatDataFileListingOptions& options, + std::vector* files); +}; + +} // namespace paimon diff --git a/src/paimon/core/table/format/format_file_listing_test.cpp b/src/paimon/core/table/format/format_file_listing_test.cpp new file mode 100644 index 00000000..8ac432b3 --- /dev/null +++ b/src/paimon/core/table/format/format_file_listing_test.cpp @@ -0,0 +1,161 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include "paimon/core/table/format/format_file_listing.h" + +#include +#include +#include +#include + +#include "gtest/gtest.h" +#include "paimon/common/utils/path_util.h" +#include "paimon/fs/file_system.h" +#include "paimon/fs/local/local_file_system.h" +#include "paimon/testing/utils/testharness.h" + +namespace paimon::test { + +namespace { + +Status WriteAt(const std::shared_ptr& file_system, const std::string& path) { + std::string parent = PathUtil::GetParentDirPath(path); + PAIMON_RETURN_NOT_OK(file_system->Mkdirs(parent)); + return file_system->WriteFile(path, "row\n", /*overwrite=*/true); +} + +/// The listed files as paths relative to `root`, sorted so the order of a listing cannot matter. +Result> ListNames(const std::shared_ptr& file_system, + const std::string& root, + const FormatDataFileListingOptions& options) { + std::vector files; + PAIMON_RETURN_NOT_OK(FormatFileListing::ListDataFiles(file_system, root, options, &files)); + std::vector names; + names.reserve(files.size()); + for (const FormatDataSplit::FileMeta& file : files) { + names.push_back(file.file_path.substr(root.size() + 1)); + } + std::sort(names.begin(), names.end()); + return names; +} + +} // namespace + +TEST(FormatFileListingTest, TestDescendsIntoPlainSubdirectories) { + std::unique_ptr dir = UniqueTestDirectory::Create(); + ASSERT_TRUE(dir); + std::shared_ptr file_system = std::make_shared(); + // `data-file.path-directory`, and other engines, put data files below the partition directory + // rather than directly in it, so stopping at the top level would miss them. + ASSERT_OK(WriteAt(file_system, dir->Str() + "/a.parquet")); + ASSERT_OK(WriteAt(file_system, dir->Str() + "/nested/b.parquet")); + ASSERT_OK(WriteAt(file_system, dir->Str() + "/nested/deeper/c.parquet")); + + ASSERT_OK_AND_ASSIGN(std::vector names, + ListNames(file_system, dir->Str(), FormatDataFileListingOptions{})); + ASSERT_EQ(names, (std::vector{"a.parquet", "nested/b.parquet", + "nested/deeper/c.parquet"})); +} + +TEST(FormatFileListingTest, TestHiddenNamesAreSkippedAndNotDescendedInto) { + std::unique_ptr dir = UniqueTestDirectory::Create(); + ASSERT_TRUE(dir); + std::shared_ptr file_system = std::make_shared(); + // A staging tree holds another job's uncommitted output under ordinary data file names, so + // only the directory above them tells the two apart. + ASSERT_OK(WriteAt(file_system, dir->Str() + "/a.parquet")); + ASSERT_OK(WriteAt(file_system, dir->Str() + "/.b.parquet.tmp")); + ASSERT_OK(WriteAt(file_system, dir->Str() + "/_temporary/c.parquet")); + ASSERT_OK(WriteAt(file_system, dir->Str() + "/.hive-staging_1/d.parquet")); + + ASSERT_OK_AND_ASSIGN(std::vector names, + ListNames(file_system, dir->Str(), FormatDataFileListingOptions{})); + ASSERT_EQ(names, (std::vector{"a.parquet"})); +} + +TEST(FormatFileListingTest, TestDefaultPartitionDirectoryIsTheOneHiddenNameThatIsContent) { + std::unique_ptr dir = UniqueTestDirectory::Create(); + ASSERT_TRUE(dir); + std::shared_ptr file_system = std::make_shared(); + // In the value-only layout a partition directory is the bare value, so a null partition is + // named `__DEFAULT_PARTITION__` - a hidden name that nonetheless holds table data. + ASSERT_OK(WriteAt(file_system, dir->Str() + "/__DEFAULT_PARTITION__/a.parquet")); + ASSERT_OK(WriteAt(file_system, dir->Str() + "/_temporary/b.parquet")); + + FormatDataFileListingOptions options; + options.partition_levels = 1; + options.only_value_in_path = true; + options.default_part_name = "__DEFAULT_PARTITION__"; + ASSERT_OK_AND_ASSIGN(std::vector names, + ListNames(file_system, dir->Str(), options)); + ASSERT_EQ(names, (std::vector{"__DEFAULT_PARTITION__/a.parquet"})); + + // With no partition level below the root, that name is a staging tree like any other. + FormatDataFileListingOptions no_partition_level = options; + no_partition_level.partition_levels = 0; + ASSERT_OK_AND_ASSIGN(std::vector without, + ListNames(file_system, dir->Str(), no_partition_level)); + ASSERT_TRUE(without.empty()); +} + +TEST(FormatFileListingTest, TestReservedDirectoriesAreSkippedOnlyWhenTheyAreMetadata) { + std::unique_ptr dir = UniqueTestDirectory::Create(); + ASSERT_TRUE(dir); + std::shared_ptr file_system = std::make_shared(); + ASSERT_OK(WriteAt(file_system, dir->Str() + "/a.parquet")); + ASSERT_OK(WriteAt(file_system, dir->Str() + "/schema/schema-0")); + + FormatDataFileListingOptions metadata_here; + metadata_here.skip_reserved_directories = true; + ASSERT_OK_AND_ASSIGN(std::vector skipped, + ListNames(file_system, dir->Str(), metadata_here)); + ASSERT_EQ(skipped, (std::vector{"a.parquet"})); + + // For a table whose schema lives in a metastore, the location is nothing but data and a + // directory of that name is a partition value or a data subdirectory. + ASSERT_OK_AND_ASSIGN(std::vector kept, + ListNames(file_system, dir->Str(), FormatDataFileListingOptions{})); + ASSERT_EQ(kept, (std::vector{"a.parquet", "schema/schema-0"})); +} + +TEST(FormatFileListingTest, TestMissingRootIsAnErrorButAVanishedSubdirectoryIsNot) { + std::unique_ptr dir = UniqueTestDirectory::Create(); + ASSERT_TRUE(dir); + std::shared_ptr file_system = std::make_shared(); + // A root that is not there means the location is wrong or the data is gone. The file systems + // here report a missing directory as an empty listing, so this only works because the root is + // asked about outright - passing it off as a table with no rows would hide a mistyped path. + std::vector files; + ASSERT_NOK(FormatFileListing::ListDataFiles(file_system, dir->Str() + "/absent", + FormatDataFileListingOptions{}, &files)); + + // A root that is a file, not a directory, is wrong in the same way. The table's own data file + // stands in for it, so nothing is left behind to turn up in the listing below. + ASSERT_OK(WriteAt(file_system, dir->Str() + "/a.parquet")); + ASSERT_NOK_WITH_MSG(FormatFileListing::ListDataFiles(file_system, dir->Str() + "/a.parquet", + FormatDataFileListingOptions{}, &files), + "is not a directory"); + + // A directory below it is another matter: it can be gone by the time the listing reaches it, + // and the rest of the listing still stands. + ASSERT_OK_AND_ASSIGN(std::vector names, + ListNames(file_system, dir->Str(), FormatDataFileListingOptions{})); + ASSERT_EQ(names, (std::vector{"a.parquet"})); +} + +} // namespace paimon::test diff --git a/src/paimon/core/table/format/format_file_naming.cpp b/src/paimon/core/table/format/format_file_naming.cpp new file mode 100644 index 00000000..98274fc5 --- /dev/null +++ b/src/paimon/core/table/format/format_file_naming.cpp @@ -0,0 +1,96 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include "paimon/core/table/format/format_file_naming.h" + +#include "fmt/format.h" +#include "paimon/common/utils/string_utils.h" +#include "paimon/common/utils/uuid.h" +#include "paimon/core/utils/partition_path_utils.h" +#include "paimon/defs.h" +#include "paimon/status.h" + +namespace paimon { + +namespace { + +/// Fails when `value` is anything other than part of a single file name. +Status CheckFileNameComponent(const std::string& value, const std::string& what) { + if (value.find('/') != std::string::npos || value.find('\\') != std::string::npos) { + return Status::Invalid( + fmt::format("{} '{}' cannot contain a path separator: it names part of one file, not " + "a path", + what, value)); + } + if (value == "." || value == ".." || value.find("..") != std::string::npos) { + return Status::Invalid(fmt::format("{} '{}' cannot contain '..'", what, value)); + } + return Status::OK(); +} + +} // namespace + +Result FormatFileNaming::Create(const std::string& extension, + const std::string& prefix) { + if (extension.empty()) { + return Status::Invalid("format table file naming requires a file extension"); + } + // Both go straight into a file name joined onto a directory, and the file is created before + // any commit sees it, so this is the only place a separator or a `..` can be stopped. + PAIMON_RETURN_NOT_OK(CheckFileNameComponent(prefix, Options::DATA_FILE_PREFIX)); + PAIMON_RETURN_NOT_OK(CheckFileNameComponent(extension, "file extension")); + // A hidden prefix would name files this table's own scan skips. + if (PartitionPathUtils::IsHiddenName(prefix)) { + return Status::Invalid( + fmt::format("{} '{}' cannot start with '_' or '.': a scan skips every file whose name " + "does", + Options::DATA_FILE_PREFIX, prefix)); + } + std::string uuid; + if (!UUID::Generate(&uuid)) { + return Status::Invalid("failed to generate uuid for format table file naming"); + } + return FormatFileNaming(uuid, extension, prefix); +} + +std::string FormatFileNaming::NextFileName() { + return fmt::format("{}{}-{}.{}", prefix_, uuid_, file_count_++, extension_); +} + +Result FormatFileNaming::NextTempFilePath() { + // A uuid of its own rather than this write's, as Java Paimon does: `_temporary` is shared, + // and a name derived from the target would collide with a retry of the same write. + std::string uuid; + if (!UUID::Generate(&uuid)) { + return Status::Invalid("failed to generate uuid for a staged format table file"); + } + return fmt::format("{}/{}{}", kTempDirName, kTempFilePrefix, uuid); +} + +bool FormatFileNaming::IsTempFilePath(const std::string& relative_path) { + const std::string directory_prefix = std::string(kTempDirName) + "/"; + if (!StringUtils::StartsWith(relative_path, directory_prefix)) { + return false; + } + const std::string name = relative_path.substr(directory_prefix.size()); + // One level below `_temporary` only: a deeper path is another job's staging tree, not ours. + return name.find('/') == std::string::npos && StringUtils::StartsWith(name, kTempFilePrefix) && + name.size() > std::string(kTempFilePrefix).size(); +} + +} // namespace paimon diff --git a/src/paimon/core/table/format/format_file_naming.h b/src/paimon/core/table/format/format_file_naming.h new file mode 100644 index 00000000..98d9136c --- /dev/null +++ b/src/paimon/core/table/format/format_file_naming.h @@ -0,0 +1,73 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#pragma once + +#include +#include + +#include "paimon/result.h" + +namespace paimon { + +/// Names the data files one format table write produces. +/// +/// `{prefix}{uuid}-{n}.{extension}`, the convention every paimon writer follows: the uuid belongs +/// to this write and the counter to its files, so two concurrent writers cannot collide. +/// +/// A file is staged under `_temporary/.tmp.{uuid}` beside where it will end up and takes its real +/// name only on commit, as Java Paimon's `RenamingTwoPhaseOutputStream` does. Both the directory +/// and the name are hidden, which is the Hive-style convention for output that is not committed +/// table data and is what a scan of this table skips. +class FormatFileNaming { + public: + static constexpr char kDefaultDataFilePrefix[] = "data-"; + /// Directory a staged file waits in, shared with every other writer of the same table. + static constexpr char kTempDirName[] = "_temporary"; + static constexpr char kTempFilePrefix[] = ".tmp."; + + /// @param extension File extension without its dot, which is the format's identifier. + /// @param prefix File name prefix, from `data-file.prefix`. It may not be hidden by the + /// `_` / `.` convention, since a scan skips every such file. + static Result Create(const std::string& extension, const std::string& prefix); + + FormatFileNaming() = default; + + /// The name the next file takes once committed. + std::string NextFileName(); + + /// Where the next file is staged, relative to the directory it will be published in. Each + /// staged name carries a uuid of its own, so two writers sharing `_temporary` cannot + /// collide. + Result NextTempFilePath(); + + /// Whether `relative_path` is one `NextTempFilePath()` could have produced. + static bool IsTempFilePath(const std::string& relative_path); + + private: + FormatFileNaming(const std::string& uuid, const std::string& extension, + const std::string& prefix) + : uuid_(uuid), extension_(extension), prefix_(prefix) {} + + std::string uuid_; + std::string extension_; + std::string prefix_ = kDefaultDataFilePrefix; + int64_t file_count_ = 0; +}; + +} // namespace paimon diff --git a/src/paimon/core/table/format/format_file_naming_test.cpp b/src/paimon/core/table/format/format_file_naming_test.cpp new file mode 100644 index 00000000..384aa2a6 --- /dev/null +++ b/src/paimon/core/table/format/format_file_naming_test.cpp @@ -0,0 +1,109 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include "paimon/core/table/format/format_file_naming.h" + +#include + +#include "gtest/gtest.h" +#include "paimon/common/utils/string_utils.h" +#include "paimon/status.h" +#include "paimon/testing/utils/testharness.h" + +namespace paimon::test { + +TEST(FormatFileNamingTest, TestNamesAreDataPrefixedAndNumbered) { + ASSERT_OK_AND_ASSIGN( + FormatFileNaming naming, + FormatFileNaming::Create("parquet", FormatFileNaming::kDefaultDataFilePrefix)); + std::string first = naming.NextFileName(); + std::string second = naming.NextFileName(); + + ASSERT_TRUE(StringUtils::StartsWith(first, "data-")); + ASSERT_TRUE(StringUtils::EndsWith(first, "-0.parquet")); + ASSERT_TRUE(StringUtils::EndsWith(second, "-1.parquet")); + // Both files of one write share its uuid. + ASSERT_EQ(first.substr(0, first.size() - std::string("-0.parquet").size()), + second.substr(0, second.size() - std::string("-1.parquet").size())); +} + +TEST(FormatFileNamingTest, TestTwoWritesDoNotCollide) { + ASSERT_OK_AND_ASSIGN( + FormatFileNaming first_write, + FormatFileNaming::Create("parquet", FormatFileNaming::kDefaultDataFilePrefix)); + ASSERT_OK_AND_ASSIGN( + FormatFileNaming second_write, + FormatFileNaming::Create("parquet", FormatFileNaming::kDefaultDataFilePrefix)); + ASSERT_NE(first_write.NextFileName(), second_write.NextFileName()); +} + +TEST(FormatFileNamingTest, TestTempPathIsAHiddenNameInATemporaryDirectory) { + ASSERT_OK_AND_ASSIGN( + FormatFileNaming naming, + FormatFileNaming::Create("parquet", FormatFileNaming::kDefaultDataFilePrefix)); + ASSERT_OK_AND_ASSIGN(std::string first, naming.NextTempFilePath()); + ASSERT_OK_AND_ASSIGN(std::string second, naming.NextTempFilePath()); + // The `_temporary` directory and the leading '.' are both hidden, which is what a scan skips, + // and are the layout Java Paimon stages under. + ASSERT_TRUE(StringUtils::StartsWith(first, "_temporary/.tmp.")) << first; + ASSERT_TRUE(FormatFileNaming::IsTempFilePath(first)); + // Each staged name carries a uuid of its own, so two writers staging into the one shared + // `_temporary` directory cannot collide. + ASSERT_NE(first, second); + + // Anything else is not a path this write staged: a plain file beside the target, a name + // outside `_temporary`, or a deeper tree another job staged into. + ASSERT_FALSE(FormatFileNaming::IsTempFilePath("data-abc-0.parquet")); + ASSERT_FALSE(FormatFileNaming::IsTempFilePath(".data-abc-0.parquet.tmp")); + ASSERT_FALSE(FormatFileNaming::IsTempFilePath("_temporary/")); + ASSERT_FALSE(FormatFileNaming::IsTempFilePath("_temporary/.tmp.")); + ASSERT_FALSE(FormatFileNaming::IsTempFilePath("_temporary/attempt_0/part-0.parquet")); + ASSERT_FALSE(FormatFileNaming::IsTempFilePath("_temporary/0/.tmp.abc")); +} + +TEST(FormatFileNamingTest, TestPrefixComesFromDataFilePrefix) { + ASSERT_OK_AND_ASSIGN(FormatFileNaming naming, FormatFileNaming::Create("parquet", "part-")); + std::string name = naming.NextFileName(); + ASSERT_TRUE(StringUtils::StartsWith(name, "part-")); + ASSERT_TRUE(StringUtils::EndsWith(name, "-0.parquet")); +} + +TEST(FormatFileNamingTest, TestRejectsAPrefixThatIsNotOneFileNameComponent) { + // The prefix goes straight into a file name that is joined onto a directory, and the file is + // created before any commit sees it - so a separator or a `..` here would put data outside the + // table with nothing left to stop it. + for (const char* prefix : + {"nested/", "../outside-", "nested/../../outside-", "a\\b", "..", "."}) { + ASSERT_NOK(FormatFileNaming::Create("parquet", prefix)) << prefix; + } + // The extension reaches the same file name, so it is held to the same rule. + ASSERT_NOK(FormatFileNaming::Create("../parquet", FormatFileNaming::kDefaultDataFilePrefix)); +} + +TEST(FormatFileNamingTest, TestRejectsHiddenPrefix) { + // A scan skips every file whose name starts with '_' or '.', so such a prefix would write + // rows that can never be read back. + ASSERT_NOK(FormatFileNaming::Create("parquet", "_data-")); + ASSERT_NOK(FormatFileNaming::Create("parquet", ".data-")); +} + +TEST(FormatFileNamingTest, TestRejectsEmptyExtension) { + ASSERT_NOK(FormatFileNaming::Create("", FormatFileNaming::kDefaultDataFilePrefix)); +} + +} // namespace paimon::test diff --git a/src/paimon/core/table/format/format_path_validation.cpp b/src/paimon/core/table/format/format_path_validation.cpp new file mode 100644 index 00000000..4d4ae343 --- /dev/null +++ b/src/paimon/core/table/format/format_path_validation.cpp @@ -0,0 +1,285 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include "paimon/core/table/format/format_path_validation.h" + +#include +#include +#include +#include +#include + +#include "fmt/format.h" +#include "paimon/common/utils/path_util.h" +#include "paimon/core/table/format/format_file_listing.h" +#include "paimon/core/utils/partition_path_utils.h" + +namespace paimon { + +namespace { + +/// A table location as a prefix of the paths below it. +struct LocationPrefix { + /// Without its trailing separator, so a location written either way compares the same. + std::string root; + /// Index in a path under the location where the first component below it starts. + size_t components_at = 0; +}; + +/// Resolves `directory` into the prefix the paths below it share, so that no two checks here can +/// disagree about what "under the table location" means. +/// +/// A location that is nothing but separators is the file system root, which is its own separator: +/// the component after it starts one character in, not two. An empty location names no directory, +/// and treating it as a prefix would make every absolute path pass. +Result ResolveLocationPrefix(const std::string& directory, const char* subject, + const std::string& what) { + size_t end = directory.size(); + while (end > 0 && directory[end - 1] == '/') { + end--; + } + if (end == 0) { + if (directory.empty()) { + return Status::Invalid(fmt::format( + "{} cannot be checked: its {} is empty, and an empty path is a prefix of nothing", + what, subject)); + } + return LocationPrefix{"/", 1}; + } + return LocationPrefix{directory.substr(0, end), end + 1}; +} + +bool IsUnderPrefix(const std::string& path, const LocationPrefix& prefix) { + if (path.size() <= prefix.components_at || + path.compare(0, prefix.root.size(), prefix.root) != 0) { + return false; + } + // The file system root is its own separator; every other location is followed by one. + return prefix.components_at == prefix.root.size() || path[prefix.root.size()] == '/'; +} + +} // namespace + +Status FormatPathValidation::ValidatePathUnderLocation(const std::string& path, + const std::string& location, + const std::string& what) { + PAIMON_ASSIGN_OR_RAISE(LocationPrefix prefix, + ResolveLocationPrefix(location, "table location", what)); + if (!IsUnderPrefix(path, prefix)) { + return Status::Invalid(fmt::format( + "{} names '{}', which is not under the table location '{}'", what, path, location)); + } + + // `/../victim` passes any prefix test and still resolves outside the table. + size_t begin = prefix.components_at; + while (begin <= path.size()) { + size_t end = path.find('/', begin); + if (end == std::string::npos) { + end = path.size(); + } + const std::string component = path.substr(begin, end - begin); + if (component.empty() || component == "." || component == "..") { + return Status::Invalid(fmt::format( + "{} names '{}', whose path does not stay inside the table location", what, path)); + } + begin = end + 1; + } + return Status::OK(); +} + +namespace { + +/// Fails when a scan would not reach `path`, whose last component names a file when `ends_in_file` +/// and a directory otherwise. One walk serves both: the distinction matters only for the last +/// component, which as a directory may be reserved or stand for a null partition. +Status ValidateComponentsAreVisible(const std::shared_ptr& table, + const std::string& path, bool ends_in_file, + const std::string& what) { + const std::vector& partition_keys = table->PartitionKeys(); + PAIMON_ASSIGN_OR_RAISE(LocationPrefix prefix, + ResolveLocationPrefix(table->Location(), "table location", what)); + const bool only_value = table->PartitionOnlyValueInPath(); + + size_t begin = prefix.components_at; + size_t level = 0; + while (begin <= path.size()) { + size_t end = path.find('/', begin); + const bool is_last = end == std::string::npos; + if (is_last) { + end = path.size(); + } + const std::string component = path.substr(begin, end - begin); + const bool is_directory = !is_last || !ends_in_file; + + // The one hidden name a scan reads: a null partition's directory in the value-only + // layout, and only where a partition directory belongs. + const bool is_default_partition_dir = is_directory && only_value && + level < partition_keys.size() && + component == table->PartitionDefaultName(); + if (PartitionPathUtils::IsHiddenName(component) && !is_default_partition_dir) { + return Status::Invalid(fmt::format( + "{} names '{}', which a scan of this table would skip: '{}' is hidden, and that is " + "how an uncommitted job marks its output", + what, path, component)); + } + // Only right below the location, and only when the schema lives there. A value-only + // partition lands here unescaped, so one named `schema` would be written over it. + if (level == 0 && is_directory && table->LocationCarriesPaimonMetadata() && + FormatFileListing::IsReservedDirectory(component)) { + return Status::Invalid(fmt::format( + "{} names '{}', where '{}' is this table's own metadata rather than data", what, + path, component)); + } + begin = end + 1; + level++; + } + return Status::OK(); +} + +} // namespace + +Status FormatPathValidation::ValidateFileIsVisible(const std::shared_ptr& table, + const std::string& file_path, + const std::string& what) { + return ValidateComponentsAreVisible(table, file_path, /*ends_in_file=*/true, what); +} + +Result FormatPathValidation::IsTableLocation(const std::shared_ptr& table, + const std::string& directory) { + const std::string what = fmt::format("table {}", table->FullName()); + PAIMON_ASSIGN_OR_RAISE(LocationPrefix location, + ResolveLocationPrefix(table->Location(), "table location", what)); + PAIMON_ASSIGN_OR_RAISE(LocationPrefix candidate, + ResolveLocationPrefix(directory, "directory", what)); + return location.root == candidate.root; +} + +Status FormatPathValidation::ValidateDirectoryIsVisible(const std::shared_ptr& table, + const std::string& directory, + const std::string& what) { + // Trailing separators go first, or the walk below sees an empty last component. + PAIMON_ASSIGN_OR_RAISE(LocationPrefix directory_prefix, + ResolveLocationPrefix(directory, "directory", what)); + return ValidateComponentsAreVisible(table, directory_prefix.root, /*ends_in_file=*/false, what); +} + +Status FormatPathValidation::ValidatePartitionKeys( + const std::shared_ptr& table, const std::map& partition, + const std::string& what) { + const std::vector& partition_keys = table->PartitionKeys(); + if (partition.size() != partition_keys.size()) { + return Status::Invalid( + fmt::format("{} carries {} partition values but table {} is partitioned by {} fields", + what, partition.size(), table->FullName(), partition_keys.size())); + } + for (const std::string& partition_key : partition_keys) { + if (partition.find(partition_key) == partition.end()) { + return Status::Invalid(fmt::format("{} does not carry a value for partition field '{}'", + what, partition_key)); + } + } + return Status::OK(); +} + +Status FormatPathValidation::ValidateFileInPartition( + const std::shared_ptr& table, const std::string& file_path, + const std::map& partition, const std::string& what) { + const std::vector& partition_keys = table->PartitionKeys(); + PAIMON_ASSIGN_OR_RAISE(LocationPrefix prefix, + ResolveLocationPrefix(table->Location(), "table location", what)); + // The directory components between the location and the file name; the leading + // `partition_keys.size()` of them are the partition directories. + std::vector components; + size_t begin = prefix.components_at; + while (begin < file_path.size()) { + size_t end = file_path.find('/', begin); + if (end == std::string::npos) { + break; + } + components.push_back(file_path.substr(begin, end - begin)); + begin = end + 1; + } + if (components.size() < partition_keys.size()) { + return Status::Invalid( + fmt::format("{} names '{}', which sits above the {} partition directories of table {}", + what, file_path, partition_keys.size(), table->FullName())); + } + + const bool only_value = table->PartitionOnlyValueInPath(); + for (size_t i = 0; i < partition_keys.size(); i++) { + const std::string& partition_key = partition_keys[i]; + std::string value; + if (only_value) { + value = PartitionPathUtils::UnescapePathName(components[i]); + } else { + std::optional> key_value = + PartitionPathUtils::ExtractPartitionKeyValue(components[i]); + if (!key_value || key_value->first != partition_key) { + return Status::Invalid( + fmt::format("{} names '{}', whose directory '{}' is not a partition of '{}'", + what, file_path, components[i], partition_key)); + } + value = key_value->second; + } + auto iter = partition.find(partition_key); + if (iter == partition.end() || iter->second != value) { + return Status::Invalid(fmt::format( + "{} sits in the '{}' partition of '{}' but claims '{}'", what, value, partition_key, + iter == partition.end() ? std::string("nothing") : iter->second)); + } + } + return Status::OK(); +} + +Result FormatPathValidation::BuildPartitionDirectory( + const std::shared_ptr& table, + const std::map& partition) { + const std::vector& partition_keys = table->PartitionKeys(); + std::vector> ordered_partition; + ordered_partition.reserve(partition_keys.size()); + for (const std::string& partition_key : partition_keys) { + auto iter = partition.find(partition_key); + if (iter == partition.end()) { + return Status::Invalid(fmt::format("no value for partition field '{}' of table {}", + partition_key, table->FullName())); + } + ordered_partition.emplace_back(partition_key, iter->second); + } + PAIMON_ASSIGN_OR_RAISE(std::string partition_path, + PartitionPathUtils::GeneratePartitionPath( + ordered_partition, table->PartitionOnlyValueInPath())); + PAIMON_ASSIGN_OR_RAISE(LocationPrefix prefix, + ResolveLocationPrefix(table->Location(), "table location", + fmt::format("table {}", table->FullName()))); + if (partition_path.empty()) { + return prefix.root; + } + const std::string directory = PathUtil::JoinPath(prefix.root, partition_path); + // A directory a scan would skip could be written but never read back, and an overwrite of it + // would clear whatever does live there. Every writer and commit derives its directory here. + std::string described; + for (const auto& [key, value] : ordered_partition) { + described += described.empty() ? "" : ", "; + described += fmt::format("{}={}", key, value); + } + PAIMON_RETURN_NOT_OK(FormatPathValidation::ValidateDirectoryIsVisible( + table, directory, fmt::format("partition {}", described))); + return directory; +} + +} // namespace paimon diff --git a/src/paimon/core/table/format/format_path_validation.h b/src/paimon/core/table/format/format_path_validation.h new file mode 100644 index 00000000..69a64ee8 --- /dev/null +++ b/src/paimon/core/table/format/format_path_validation.h @@ -0,0 +1,89 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#pragma once + +#include +#include +#include + +#include "paimon/result.h" +#include "paimon/status.h" +#include "paimon/table/format/format_table.h" + +namespace paimon { + +/// Checks that a path a caller handed over really names something of this format table. +/// +/// A split and a commit message both come back through an interface that takes the base type, so +/// what arrives may belong to another plan, another table or a plan made before the files moved, +/// and nothing further down re-checks the paths they name. +class FormatPathValidation { + public: + FormatPathValidation() = delete; + ~FormatPathValidation() = delete; + + /// Fails when `path` is not a file inside `location`. By path component, not by string + /// prefix: `/../victim` starts with the location and still resolves outside it. + /// Only the path text is checked, so a symbolic link pointing out of the table is not caught. + static Status ValidatePathUnderLocation(const std::string& path, const std::string& location, + const std::string& what); + + /// The directory a partition's files belong in. Only for building a path to write or clear; + /// to check one that already exists use `ValidateFileInPartition()`, since another engine may + /// spell the same value differently. + static Result BuildPartitionDirectory( + const std::shared_ptr& table, + const std::map& partition); + + /// Fails when `file_path` does not sit in the directory `partition` names, which would + /// publish rows under a partition they never had or clear the wrong one on an overwrite. + /// + /// Compared on the values the directory names spell out, not on the directory string: a value + /// can be escaped more than one way, and `100%` and `100%25` name the same one. Levels below + /// the partition are allowed. + static Status ValidateFileInPartition(const std::shared_ptr& table, + const std::string& file_path, + const std::map& partition, + const std::string& what); + + /// Fails when `file_path` is a file a scan of this table would never return: a hidden `_` / + /// `.` name, or this table's own `schema` or `branch` directory. A split that did not come + /// from this scan never went through that listing, so the same rules run here. + static Status ValidateFileIsVisible(const std::shared_ptr& table, + const std::string& file_path, const std::string& what); + + /// Whether `directory` is the table's own location, whichever way either was written. It + /// decides whether `schema` and `branch` below it are metadata or data, so a trailing + /// separator must not make the two compare different. + static Result IsTableLocation(const std::shared_ptr& table, + const std::string& directory); + + /// `ValidateFileIsVisible()` for a path ending in a directory. The last component is what + /// makes it a separate call: in the value-only layout a partition named `schema` would be + /// written where a file system catalog keeps the table's own metadata. + static Status ValidateDirectoryIsVisible(const std::shared_ptr& table, + const std::string& directory, const std::string& what); + + /// Fails when `partition` does not name exactly the table's partition keys. + static Status ValidatePartitionKeys(const std::shared_ptr& table, + const std::map& partition, + const std::string& what); +}; + +} // namespace paimon diff --git a/src/paimon/core/table/format/format_table.cpp b/src/paimon/core/table/format/format_table.cpp new file mode 100644 index 00000000..8f9cbf10 --- /dev/null +++ b/src/paimon/core/table/format/format_table.cpp @@ -0,0 +1,196 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include "paimon/table/format/format_table.h" + +#include + +#include "fmt/format.h" +#include "paimon/common/utils/checked_cast.h" +#include "paimon/common/utils/options_utils.h" +#include "paimon/common/utils/string_utils.h" +#include "paimon/core/core_options.h" +#include "paimon/core/options/table_type.h" +#include "paimon/core/schema/schema_manager.h" +#include "paimon/core/schema/schema_validation.h" +#include "paimon/core/schema/table_schema.h" +#include "paimon/defs.h" +#include "paimon/fs/file_system.h" + +namespace paimon { + +namespace { +constexpr char kDefaultFileFormat[] = "parquet"; +} // namespace + +FormatTable::FormatTable(const std::shared_ptr& file_system, + const std::string& location, const Identifier& identifier, + const std::shared_ptr& schema, + const std::map& options, Format format, + const std::string& file_compression, + const std::string& partition_default_name, + bool partition_only_value_in_path, bool location_carries_paimon_metadata) + : file_system_(file_system), + location_(location), + identifier_(identifier), + schema_(schema), + options_(options), + format_(format), + file_compression_(file_compression), + partition_default_name_(partition_default_name), + partition_only_value_in_path_(partition_only_value_in_path), + location_carries_paimon_metadata_(location_carries_paimon_metadata) {} + +FormatTable::~FormatTable() = default; + +Result FormatTable::ParseFormat(const std::string& file_format) { + std::string normalized = StringUtils::ToLowerCase(file_format); + if (normalized == "parquet") { + return Format::PARQUET; + } + if (normalized == "orc") { + return Format::ORC; + } + if (normalized == "csv" || normalized == "text" || normalized == "json" || + normalized == "mosaic") { + return Status::NotImplemented( + fmt::format("format table file format '{}' is not supported by paimon-cpp yet. " + "Supported formats: parquet, orc", + file_format)); + } + return Status::Invalid(fmt::format( + "format table unsupported file format: {}. Supported formats: parquet, orc", file_format)); +} + +std::string FormatTable::FormatToString(Format format) { + switch (format) { + case Format::PARQUET: + return "parquet"; + case Format::ORC: + return "orc"; + } + return "unknown"; +} + +Result> FormatTable::Create( + const std::shared_ptr& file_system, const std::string& table_path, + const Identifier& identifier, const std::map& dynamic_options) { + if (file_system == nullptr) { + return Status::Invalid("format table requires a file system"); + } + if (table_path.empty()) { + return Status::Invalid( + fmt::format("format table {} requires a location", identifier.GetFullName())); + } + PAIMON_ASSIGN_OR_RAISE(bool exist, file_system->Exists(table_path)); + if (!exist) { + return Status::NotExist(fmt::format("{} not exist", identifier.ToString())); + } + SchemaManager schema_manager(file_system, table_path); + PAIMON_ASSIGN_OR_RAISE(std::optional> latest_schema, + schema_manager.Latest()); + if (!latest_schema) { + return Status::NotExist( + fmt::format("load table schema for {} failed", identifier.ToString())); + } + // The schema was just read from under the table path, so that is where the metadata lives. + return Create(file_system, table_path, identifier, + checked_pointer_cast(*latest_schema), + /*location_carries_paimon_metadata=*/true, dynamic_options); +} + +Result> FormatTable::Create( + const std::shared_ptr& file_system, const std::string& location, + const Identifier& identifier, const std::shared_ptr& schema, + bool location_carries_paimon_metadata, + const std::map& dynamic_options) { + if (file_system == nullptr) { + return Status::Invalid("format table requires a file system"); + } + if (location.empty()) { + // Every path is checked against the location, and an empty one is a prefix of nothing. + return Status::Invalid( + fmt::format("format table {} requires a location", identifier.GetFullName())); + } + if (schema == nullptr) { + return Status::Invalid("format table requires a schema"); + } + // The table type comes from the schema alone. `type` is structural: letting an option given + // at the call override it would let one read or write open a managed table as a format table, + // or hide a format table. + PAIMON_ASSIGN_OR_RAISE(bool is_format_table, TableTypeDefine::IsFormatTable(schema->Options())); + if (!is_format_table) { + return Status::Invalid( + fmt::format("table {} is not a format table, its '{}' option is not " + "'{}'", + identifier.GetFullName(), Options::TYPE, TableTypeDefine::kFormatTable)); + } + // The runtime options: what the schema stored, with anything given at the call on top, the + // precedence every context builder promises. `type` was read above and is not one of them. + std::map options = schema->Options(); + for (const auto& [key, value] : dynamic_options) { + if (key == Options::TYPE) { + continue; + } + options[key] = value; + } + + // The checks creation would have run, again: this schema may have come from a rest catalog or + // straight from a caller. The message names the table and keeps the original status code. + // + // Against the merged options rather than the schema's own, so that an option this refuses is + // refused wherever it comes from instead of being dropped in silence. + Status valid = SchemaValidation::ValidateGenericDataSchema(*schema); + if (valid.ok()) { + valid = SchemaValidation::ValidateFormatTableSchema(*schema, options, file_system); + } + if (!valid.ok()) { + return Status(valid.code(), fmt::format("cannot open format table {}: {}", + identifier.GetFullName(), valid.message())); + } + + // Before `CoreOptions`, which resolves `file.format` through the factories and would fail + // with a missing-factory error where this names the format. + PAIMON_ASSIGN_OR_RAISE(std::string file_format, + OptionsUtils::GetValueFromMap(options, Options::FILE_FORMAT, + kDefaultFileFormat)); + PAIMON_ASSIGN_OR_RAISE(Format format, ParseFormat(file_format)); + + // Everything else through `CoreOptions`, so an option means what it does on the managed + // table path and a default lives in one place. + PAIMON_ASSIGN_OR_RAISE(CoreOptions core_options, CoreOptions::FromMap(options, file_system)); + + return std::shared_ptr(new FormatTable( + file_system, location, identifier, schema, options, format, + core_options.FormatTableFileCompression(), core_options.GetPartitionDefaultName(), + core_options.FormatTablePartitionOnlyValueInPath(), location_carries_paimon_metadata)); +} + +const std::vector& FormatTable::PartitionKeys() const { + return schema_->PartitionKeys(); +} + +std::string FormatTable::FullName() const { + return identifier_.GetFullName(); +} + +Result> FormatTable::GetArrowSchema() const { + return schema_->GetArrowSchema(); +} + +} // namespace paimon diff --git a/src/paimon/core/table/format/format_table_commit.cpp b/src/paimon/core/table/format/format_table_commit.cpp new file mode 100644 index 00000000..d726c0d6 --- /dev/null +++ b/src/paimon/core/table/format/format_table_commit.cpp @@ -0,0 +1,336 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include "paimon/core/table/format/format_table_commit.h" + +#include +#include +#include +#include +#include +#include + +#include "fmt/format.h" +#include "fmt/ranges.h" +#include "paimon/common/utils/path_util.h" +#include "paimon/common/utils/string_utils.h" +#include "paimon/core/table/format/format_file_listing.h" +#include "paimon/core/table/format/format_file_naming.h" +#include "paimon/core/table/format/format_path_validation.h" +#include "paimon/core/utils/partition_path_utils.h" +#include "paimon/fs/file_system.h" +#include "paimon/logging.h" + +namespace paimon { + +namespace { + +Logger* CommitLogger() { + static std::unique_ptr logger = Logger::GetLogger("FormatTableCommit"); + return logger.get(); +} + +/// Fails when a commit message does not describe a file of this table. +/// +/// A commit takes the messages a caller held on to, and committing one renames a path while an +/// overwrite clears the directory around it. Only the shape of a message is checked, not who +/// produced it. +Status ValidateCommitMessage(const FormatCommitMessage& message, + const std::shared_ptr& table, + const std::map& static_partition) { + const std::string what = fmt::format("commit message {}", message.ToString()); + PAIMON_RETURN_NOT_OK(FormatPathValidation::ValidatePathUnderLocation(message.file_path, + table->Location(), what)); + PAIMON_RETURN_NOT_OK(FormatPathValidation::ValidatePathUnderLocation(message.temp_file_path, + table->Location(), what)); + + // `/_temporary/.tmp.`, which a scan skips and which is what Java + // Paimon's `RenamingTwoPhaseOutputStream` stages under. + const std::string publish_directory = PathUtil::GetParentDirPath(message.file_path); + const std::string directory_prefix = publish_directory + "/"; + if (!StringUtils::StartsWith(message.temp_file_path, directory_prefix) || + !FormatFileNaming::IsTempFilePath(message.temp_file_path.substr(directory_prefix.size()))) { + return Status::Invalid(fmt::format( + "{} does not stage its file under '{}/{}...' beside where it will be published, so it " + "may already be visible or belong to another directory", + what, FormatFileNaming::kTempDirName, FormatFileNaming::kTempFilePrefix)); + } + // Otherwise an overwrite could clear the old data and publish a file nothing can ever read. + PAIMON_RETURN_NOT_OK( + FormatPathValidation::ValidateFileIsVisible(table, message.file_path, what)); + if (message.record_count < 0 || message.file_size < 0) { + return Status::Invalid(fmt::format("{} reports a negative row count or file size", what)); + } + + // Otherwise an overwrite would clear the wrong partition. + PAIMON_RETURN_NOT_OK( + FormatPathValidation::ValidatePartitionKeys(table, message.partition, what)); + PAIMON_RETURN_NOT_OK(FormatPathValidation::ValidateFileInPartition(table, message.file_path, + message.partition, what)); + + // Otherwise the file would be published into a partition this commit never cleared. + for (const auto& [key, value] : static_partition) { + auto iter = message.partition.find(key); + if (iter == message.partition.end() || iter->second != value) { + return Status::Invalid(fmt::format( + "{} is not in the static partition '{}={}' this commit writes", what, key, value)); + } + } + return Status::OK(); +} + +/// Fails when `static_partition` cannot name a directory of this table. The keys must be a prefix +/// of the partition keys, since a partition directory nests below the one before it. +Status ValidateStaticPartition(const std::map& static_partition, + const std::vector& partition_keys, + const std::string& table_name) { + if (static_partition.empty()) { + return Status::OK(); + } + if (partition_keys.empty()) { + return Status::Invalid(fmt::format( + "format table {} is not partitioned, so a static partition names nothing", table_name)); + } + for (const auto& entry : static_partition) { + const std::string& key = entry.first; + if (std::find(partition_keys.begin(), partition_keys.end(), key) == partition_keys.end()) { + return Status::Invalid( + fmt::format("'{}' is not a partition key of format table {}", key, table_name)); + } + } + bool missing_leading_key = false; + for (const std::string& partition_key : partition_keys) { + const bool named = static_partition.find(partition_key) != static_partition.end(); + if (named && missing_leading_key) { + return Status::Invalid( + fmt::format("static partition column '{}' of format table {} cannot be given " + "without the partition columns it nests under", + partition_key, table_name)); + } + if (!named) { + missing_leading_key = true; + } + } + return Status::OK(); +} + +} // namespace + +std::string FormatCommitMessage::ToString() const { + return fmt::format( + "FormatCommitMessage{{file_path={}, temp_file_path={}, partition={}, record_count={}, " + "file_size={}}}", + file_path, temp_file_path, partition, record_count, file_size); +} + +FormatTableCommit::FormatTableCommit(const std::shared_ptr& table, bool overwrite, + const std::map& static_partition) + : table_(table), overwrite_(overwrite), static_partition_(static_partition) {} + +FormatTableCommit::~FormatTableCommit() = default; + +Result> FormatTableCommit::Create( + const std::shared_ptr& table, bool overwrite, + const std::map& static_partition) { + if (table == nullptr) { + return Status::Invalid("format table commit requires a table"); + } + PAIMON_RETURN_NOT_OK( + ValidateStaticPartition(static_partition, table->PartitionKeys(), table->FullName())); + return std::unique_ptr( + new FormatTableCommit(table, overwrite, static_partition)); +} + +Status FormatTableCommit::DeletePreviousDataFiles(const std::string& directory, + int32_t partition_levels) const { + std::shared_ptr file_system = table_->GetFileSystem(); + FormatDataFileListingOptions listing; + listing.partition_levels = partition_levels; + listing.only_value_in_path = table_->PartitionOnlyValueInPath(); + listing.default_part_name = table_->PartitionDefaultName(); + // Only right at the location are `schema` and `branch` metadata; below it they are data. + PAIMON_ASSIGN_OR_RAISE(bool at_location, + FormatPathValidation::IsTableLocation(table_, directory)); + listing.skip_reserved_directories = at_location && table_->LocationCarriesPaimonMetadata(); + std::vector files; + // Committed data files only: a staging directory holds another writer's uncommitted output. + PAIMON_RETURN_NOT_OK(FormatFileListing::ListDataFiles(file_system, directory, listing, &files)); + for (const FormatDataSplit::FileMeta& file : files) { + Status status = file_system->Delete(file.file_path, /*recursive=*/false); + if (!status.ok() && !status.IsNotExist()) { + return status; + } + } + return Status::OK(); +} + +Status FormatTableCommit::Commit(const std::vector& commit_messages) { + Status status = CommitImpl(commit_messages); + if (!status.ok()) { + // The write has already prepared its commit, so nothing else will clean up what is still + // staged. `Abort()` logs its own failures and returns OK today; the status is still read. + Status abort_status = Abort(commit_messages); + if (!abort_status.ok()) { + PAIMON_LOG_WARN(CommitLogger(), "Failed to clean up table %s after a failed commit: %s", + table_->FullName().c_str(), abort_status.ToString().c_str()); + } + } + return status; +} + +Status FormatTableCommit::CommitImpl(const std::vector& commit_messages) { + std::shared_ptr file_system = table_->GetFileSystem(); + const std::vector& partition_keys = table_->PartitionKeys(); + // An overwrite deletes committed data, so what it was asked to replace is worth logging even + // when it succeeds, as the managed table commit does at the same level. + PAIMON_LOG_INFO(CommitLogger(), "Ready to %s %zu messages to format table %s", + overwrite_ ? "overwrite with" : "commit", commit_messages.size(), + table_->FullName().c_str()); + + // Before anything is renamed or deleted: an overwrite clears the directory a message names. + for (const FormatCommitMessage& message : commit_messages) { + PAIMON_RETURN_NOT_OK(ValidateCommitMessage(message, table_, static_partition_)); + } + + // Every message is checked against what is on disk before anything moves. An overwrite makes + // this critical: it clears the old data first, so a staged file that turns out to be missing + // would leave the table with the old rows gone and the new ones never arriving. + std::set targets; + for (const FormatCommitMessage& message : commit_messages) { + if (!targets.insert(message.file_path).second) { + return Status::Invalid(fmt::format( + "two commit messages would publish {}, so one would overwrite the other", + message.file_path)); + } + Result staged = file_system->GetFileStatus(message.temp_file_path); + if (!staged.ok()) { + // `Invalid` whatever the file system said, since the fault is the message; its text + // is kept all the same. + return Status::Invalid(fmt::format("the staged file {} cannot be read: {}", + message.temp_file_path, staged.status().ToString())); + } + // `rename` moves a directory as readily as a file. + if (staged.value().IsDir()) { + return Status::Invalid(fmt::format("the staged path {} is a directory, not a file", + message.temp_file_path)); + } + if (message.file_size != staged.value().GetLen()) { + return Status::Invalid( + fmt::format("the staged file {} is {} bytes but the commit message says {}", + message.temp_file_path, staged.value().GetLen(), message.file_size)); + } + } + + // What an overwrite replaces is cleared first: it removes committed files only, and this + // commit's own are still hidden, so neither can take the other out. + if (!static_partition_.empty()) { + // The spec names the leading keys in order, so the path may be a prefix with the + // partitions of the unnamed keys below it. + std::vector> ordered_partition; + ordered_partition.reserve(static_partition_.size()); + for (const std::string& partition_key : partition_keys) { + auto iter = static_partition_.find(partition_key); + if (iter == static_partition_.end()) { + break; + } + ordered_partition.emplace_back(partition_key, iter->second); + } + PAIMON_ASSIGN_OR_RAISE(std::string partition_path, + PartitionPathUtils::GeneratePartitionPath( + ordered_partition, table_->PartitionOnlyValueInPath())); + std::string directory = PathUtil::JoinPath(table_->Location(), partition_path); + // The spec may name a partition no message covers: an overwrite of a directory a scan + // skips would clear files that are not this table's data. + PAIMON_RETURN_NOT_OK(FormatPathValidation::ValidateDirectoryIsVisible(table_, directory, + "static partition")); + PAIMON_ASSIGN_OR_RAISE(bool exists, file_system->Exists(directory)); + if (!exists) { + // Nothing to clear, but created regardless: an overwrite leaves an empty partition + // behind rather than removing it from the table. + PAIMON_RETURN_NOT_OK(file_system->Mkdirs(directory)); + } else if (overwrite_) { + PAIMON_RETURN_NOT_OK(DeletePreviousDataFiles( + directory, static_cast(partition_keys.size() - ordered_partition.size()))); + } + } else if (overwrite_) { + // The directory the message's partition names, not the file's own parent. A message may + // name a file below its partition directory - `ValidateFileInPartition()` allows that, and + // `data-file.path-directory` or another engine's layout puts files there - and an + // overwrite replaces everything the partition holds, not just the subdirectory this + // commit's file landed in. The partition and the path were checked against each other + // above, so either would name the same partition; only this one names all of it. + std::set directories; + for (const FormatCommitMessage& message : commit_messages) { + PAIMON_ASSIGN_OR_RAISE( + std::string directory, + FormatPathValidation::BuildPartitionDirectory(table_, message.partition)); + directories.insert(std::move(directory)); + } + for (const std::string& directory : directories) { + // A complete partition directory, so no partition level is left below it. + PAIMON_RETURN_NOT_OK(DeletePreviousDataFiles(directory, /*partition_levels=*/0)); + } + } + + std::vector published; + published.reserve(commit_messages.size()); + for (const FormatCommitMessage& message : commit_messages) { + Status status = file_system->Rename(message.temp_file_path, message.file_path); + if (!status.ok()) { + // Take the published files back, so the table holds all of this write or none of it. + for (const std::string& file_path : published) { + Status rollback = file_system->Delete(file_path, /*recursive=*/false); + if (!rollback.ok()) { + PAIMON_LOG_WARN(CommitLogger(), "Failed to take back the published file %s: %s", + file_path.c_str(), rollback.ToString().c_str()); + } + } + return Status::IOError(fmt::format("failed to commit {} of format table {}: {}", + message.ToString(), table_->FullName(), + status.ToString())); + } + published.push_back(message.file_path); + } + return Status::OK(); +} + +Status FormatTableCommit::Abort(const std::vector& commit_messages) { + std::shared_ptr file_system = table_->GetFileSystem(); + // Best effort: a file that cannot be removed stays behind under its hidden name, where a + // scan ignores it. + for (const FormatCommitMessage& message : commit_messages) { + // Returning at the first bad one would strand the staged files of the good ones. + Status valid = ValidateCommitMessage(message, table_, static_partition_); + if (!valid.ok()) { + // Structure and location are all this can check, so the wording claims no more. + PAIMON_LOG_WARN(CommitLogger(), + "Refusing to discard a file that does not describe this table: %s", + valid.ToString().c_str()); + continue; + } + Status status = file_system->Delete(message.temp_file_path, /*recursive=*/false); + // Already gone is the expected case after a rollback took the file back. + if (!status.ok() && !status.IsNotExist()) { + PAIMON_LOG_WARN(CommitLogger(), "Failed to discard the staged file %s: %s", + message.temp_file_path.c_str(), status.ToString().c_str()); + } + } + return Status::OK(); +} + +} // namespace paimon diff --git a/src/paimon/core/table/format/format_table_commit.h b/src/paimon/core/table/format/format_table_commit.h new file mode 100644 index 00000000..c227cc39 --- /dev/null +++ b/src/paimon/core/table/format/format_table_commit.h @@ -0,0 +1,104 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#pragma once + +#include +#include +#include +#include +#include + +#include "paimon/core/table/format/format_commit_message.h" +#include "paimon/result.h" +#include "paimon/status.h" +#include "paimon/table/format/format_table.h" + +namespace paimon { + +/// Publishes the files a `FormatTableWrite` produced, by renaming each out of the `_temporary` +/// directory it was staged in. +/// +/// A directory has no metadata to switch, so a commit is not atomic across files: it renames them +/// one at a time, and a reader scanning midway sees the ones renamed so far. Each rename does +/// guarantee that a file becomes visible whole, never half-written. +/// +/// A commit that fails partway tries to remove the files it had already renamed, on a best-effort +/// basis: a file that cannot be removed is reported in the log and stays. An overwriting commit is +/// further limited - the data it replaces is deleted before the new files are published and cannot +/// be brought back, so a failure there leaves the table without the replaced data. +/// +/// Only the messages this job's own writers produced may be passed in. The checks here can tell +/// that a message's path belongs to this table, sits in the partition it declares, and is staged +/// rather than already published - not whose staged file it is. +/// +/// Not thread-safe. Separate commits may add to one table at once, each publishing only the files +/// its own messages name; two overwriting commits over the same directory race, since an overwrite +/// clears everything committed there before publishing anything. +class FormatTableCommit { + public: + /// @param table Table to commit to. + /// @param overwrite Whether the commit replaces the data already in the directories it writes + /// to, instead of adding to it. Without a static partition that means every partition + /// the commit touches; with one it means the partitions that spec covers, whether or + /// not this commit wrote to them. + /// @param static_partition Partition the commit writes to, keyed by partition field name. It + /// may name only the leading partition keys, in which case it stands for every + /// partition below that prefix. Empty means the partitions are whatever the written + /// files say they are. + static Result> Create( + const std::shared_ptr& table, bool overwrite, + const std::map& static_partition); + + ~FormatTableCommit(); + + /// Renames every written file into place, first clearing what it replaces when the commit + /// overwrites. + Status Commit(const std::vector& commit_messages); + + /// Removes the staged files of `commit_messages` instead of publishing them, on a + /// best-effort basis. + /// + /// It undoes a commit that never happened, not one that did: a file already renamed into + /// place is no longer staged and stays where it is. + /// + /// It never fails. A message that is refused, or a file that cannot be removed, does not stop + /// the remaining messages from being cleaned up, so the log is the only signal that a cleanup + /// did not fully succeed. + Status Abort(const std::vector& commit_messages); + + private: + FormatTableCommit(const std::shared_ptr& table, bool overwrite, + const std::map& static_partition); + + /// The body of `Commit()`, so that every failure in it is followed by the same cleanup. + Status CommitImpl(const std::vector& commit_messages); + + /// Deletes the committed data files under `directory`, leaving what another writer has staged + /// there untouched. + /// + /// @param partition_levels How many directory levels below `directory` still hold partition + /// directories, which a static partition naming only the leading keys leaves behind. + Status DeletePreviousDataFiles(const std::string& directory, int32_t partition_levels) const; + + std::shared_ptr table_; + bool overwrite_ = false; + std::map static_partition_; +}; + +} // namespace paimon diff --git a/src/paimon/core/table/format/format_table_file_store_commit.cpp b/src/paimon/core/table/format/format_table_file_store_commit.cpp new file mode 100644 index 00000000..3e53c526 --- /dev/null +++ b/src/paimon/core/table/format/format_table_file_store_commit.cpp @@ -0,0 +1,174 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include "paimon/core/table/format/format_table_file_store_commit.h" + +#include "fmt/format.h" +#include "paimon/common/metrics/metrics_impl.h" +#include "paimon/core/table/format/format_commit_message.h" +#include "paimon/core/table/format/format_table_commit.h" +#include "paimon/table/format/format_table.h" + +namespace paimon { + +namespace { + +/// The refusal the snapshot half of `FileStoreCommit` returns, so a caller reading one knows to +/// look at the table type rather than at its own arguments. +Status UnsupportedFormatTableOperation(const char* what) { + return Status::NotImplemented( + fmt::format("a format table has no snapshots or manifests, so {}", what)); +} + +} // namespace + +FormatTableFileStoreCommit::FormatTableFileStoreCommit(const std::shared_ptr& table) + : table_(table) {} + +FormatTableFileStoreCommit::~FormatTableFileStoreCommit() = default; + +Result> FormatTableFileStoreCommit::Create( + const std::shared_ptr& table) { + if (table == nullptr) { + return Status::Invalid("format table commit requires a table"); + } + return std::unique_ptr(new FormatTableFileStoreCommit(table)); +} + +Result> FormatTableFileStoreCommit::ToFormatMessages( + const std::vector>& commit_messages) { + std::vector messages; + messages.reserve(commit_messages.size()); + for (const std::shared_ptr& commit_message : commit_messages) { + auto message = std::dynamic_pointer_cast(commit_message); + if (message == nullptr) { + return Status::Invalid( + "a format table commit takes the messages a format table write produced; this one " + "describes files to record in a manifest"); + } + messages.push_back(*message); + } + return messages; +} + +Status FormatTableFileStoreCommit::Commit( + const std::vector>& commit_messages, int64_t commit_identifier, + std::optional watermark) { + if (commit_identifier != BATCH_WRITE_COMMIT_IDENTIFIER) { + return UnsupportedFormatTableOperation("a commit identifier has nowhere to be recorded"); + } + if (watermark) { + return UnsupportedFormatTableOperation("a watermark has nowhere to be recorded"); + } + PAIMON_ASSIGN_OR_RAISE(std::vector messages, + ToFormatMessages(commit_messages)); + PAIMON_ASSIGN_OR_RAISE( + std::unique_ptr commit, + FormatTableCommit::Create(table_, /*overwrite=*/false, /*static_partition=*/{})); + return commit->Commit(messages); +} + +Status FormatTableFileStoreCommit::Overwrite( + const std::map& partition, + const std::vector>& commit_messages, int64_t commit_identifier, + std::optional watermark) { + if (commit_identifier != BATCH_WRITE_COMMIT_IDENTIFIER) { + return UnsupportedFormatTableOperation("a commit identifier has nowhere to be recorded"); + } + if (watermark) { + return UnsupportedFormatTableOperation("a watermark has nowhere to be recorded"); + } + PAIMON_ASSIGN_OR_RAISE(std::vector messages, + ToFormatMessages(commit_messages)); + PAIMON_ASSIGN_OR_RAISE(std::unique_ptr commit, + FormatTableCommit::Create(table_, /*overwrite=*/true, partition)); + return commit->Commit(messages); +} + +Status FormatTableFileStoreCommit::Abort( + const std::vector>& commit_messages) { + PAIMON_ASSIGN_OR_RAISE(std::vector messages, + ToFormatMessages(commit_messages)); + PAIMON_ASSIGN_OR_RAISE( + std::unique_ptr commit, + FormatTableCommit::Create(table_, /*overwrite=*/false, /*static_partition=*/{})); + return commit->Abort(messages); +} + +Result FormatTableFileStoreCommit::CommitWithProgress( + const std::vector& realtime_commits, int64_t commit_identifier, + std::optional watermark) { + return UnsupportedFormatTableOperation("it has no real-time offsets to publish with a commit"); +} + +Result FormatTableFileStoreCommit::FilterAndCommit( + const std::map>>& + commit_identifier_and_messages, + std::optional watermark) { + return UnsupportedFormatTableOperation( + "nothing records which commit identifiers have already been committed"); +} + +Result FormatTableFileStoreCommit::FilterAndOverwrite( + const std::map& partition, + const std::vector>& commit_messages, int64_t commit_identifier, + std::optional watermark) { + return UnsupportedFormatTableOperation( + "nothing records which commit identifiers have already been committed"); +} + +Result FormatTableFileStoreCommit::GetLastCommitTableRequest() { + return UnsupportedFormatTableOperation( + "it commits by renaming files rather than through a rest catalog"); +} + +Result FormatTableFileStoreCommit::Expire() { + return UnsupportedFormatTableOperation("there are no snapshots to expire"); +} + +Status FormatTableFileStoreCommit::DropPartition( + const std::vector>& partitions, int64_t commit_identifier) { + // Not refused for lack of snapshots but simply missing: dropping a partition means removing + // its directory, which nothing here does yet. An overwrite of that partition with no messages + // empties it, which is as far as this goes today. + return Status::NotImplemented( + "dropping a partition of a format table is not implemented yet; an overwrite of it with " + "no commit messages empties it instead"); +} + +Status FormatTableFileStoreCommit::TruncateTable(int64_t commit_identifier) { + return Status::NotImplemented("emptying a format table is not implemented yet"); +} + +Result FormatTableFileStoreCommit::RollbackToAsLatest(int64_t target_snapshot_id) { + return UnsupportedFormatTableOperation("there is no snapshot to roll back to"); +} + +FileStoreCommit& FormatTableFileStoreCommit::RowIdCheckConflict( + std::optional row_id_check_from_snapshot) { + // The interface returns a reference and cannot report a refusal, so this is the one call that + // has to be a no-op. A format table records no row ids, so there is no conflict to check. + return *this; +} + +std::shared_ptr FormatTableFileStoreCommit::GetCommitMetrics() const { + // Empty rather than null, as on the write side's `GetMetrics()`. + return std::make_shared(); +} + +} // namespace paimon diff --git a/src/paimon/core/table/format/format_table_file_store_commit.h b/src/paimon/core/table/format/format_table_file_store_commit.h new file mode 100644 index 00000000..3a440f97 --- /dev/null +++ b/src/paimon/core/table/format/format_table_file_store_commit.h @@ -0,0 +1,109 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#pragma once + +#include +#include +#include +#include +#include +#include + +#include "paimon/core/table/format/format_commit_message.h" +#include "paimon/defs.h" +#include "paimon/file_store_commit.h" +#include "paimon/result.h" +#include "paimon/status.h" + +namespace paimon { + +class FormatTable; +class Metrics; + +/// Commits a format table through the `FileStoreCommit` interface, so that a caller holding a +/// table path commits it the way it commits any other table. Java Paimon does the same through +/// `FormatTable.newBatchWriteBuilder()`. +/// +/// Most of `FileStoreCommit` is about snapshots and manifests, which a format table has none of: +/// expiring them, rolling back to one, and filtering by a commit identifier recorded in one all +/// refer to state this table does not keep. Each is refused rather than quietly doing nothing, so +/// a caller moving between table types finds out at the call rather than from a table that did +/// not change. `RowIdCheckConflict()` is the one exception, since it returns a reference and has +/// no way to report a refusal. +/// +/// What is left is what a directory of files can do: `Commit()`, `Overwrite()` and `Abort()`. +class FormatTableFileStoreCommit : public FileStoreCommit { + public: + static Result> Create( + const std::shared_ptr& table); + + ~FormatTableFileStoreCommit() override; + + Status Commit(const std::vector>& commit_messages, + int64_t commit_identifier = BATCH_WRITE_COMMIT_IDENTIFIER, + std::optional watermark = std::nullopt) override; + + Status Overwrite(const std::map& partition, + const std::vector>& commit_messages, + int64_t commit_identifier, + std::optional watermark = std::nullopt) override; + + Status Abort(const std::vector>& commit_messages) override; + + Result CommitWithProgress(const std::vector& realtime_commits, + int64_t commit_identifier, + std::optional watermark) override; + + Result FilterAndCommit( + const std::map>>& + commit_identifier_and_messages, + std::optional watermark = std::nullopt) override; + + Result FilterAndOverwrite( + const std::map& partition, + const std::vector>& commit_messages, + int64_t commit_identifier, std::optional watermark = std::nullopt) override; + + Result GetLastCommitTableRequest() override; + + Result Expire() override; + + Status DropPartition(const std::vector>& partitions, + int64_t commit_identifier) override; + + Status TruncateTable(int64_t commit_identifier) override; + + Result RollbackToAsLatest(int64_t target_snapshot_id) override; + + FileStoreCommit& RowIdCheckConflict(std::optional row_id_check_from_snapshot) override; + + std::shared_ptr GetCommitMetrics() const override; + + private: + explicit FormatTableFileStoreCommit(const std::shared_ptr& table); + + /// The messages a format table commit takes, or a refusal when one of them belongs to another + /// table type. Nothing is published until every message has been recognised. + static Result> ToFormatMessages( + const std::vector>& commit_messages); + + std::shared_ptr table_; +}; + +} // namespace paimon diff --git a/src/paimon/core/table/format/format_table_file_store_write.cpp b/src/paimon/core/table/format/format_table_file_store_write.cpp new file mode 100644 index 00000000..a4829637 --- /dev/null +++ b/src/paimon/core/table/format/format_table_file_store_write.cpp @@ -0,0 +1,96 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include "paimon/core/table/format/format_table_file_store_write.h" + +#include + +#include "paimon/common/metrics/metrics_impl.h" +#include "paimon/core/table/format/format_commit_message.h" +#include "paimon/core/table/format/format_table_write.h" +#include "paimon/table/format/format_table.h" + +namespace paimon { + +FormatTableFileStoreWrite::FormatTableFileStoreWrite(std::unique_ptr&& write) + : write_(std::move(write)) {} + +FormatTableFileStoreWrite::~FormatTableFileStoreWrite() = default; + +Result> FormatTableFileStoreWrite::Create( + const std::shared_ptr& table, const std::shared_ptr& pool) { + PAIMON_ASSIGN_OR_RAISE(std::unique_ptr write, + FormatTableWrite::Create(table, pool)); + return std::unique_ptr( + new FormatTableFileStoreWrite(std::move(write))); +} + +Status FormatTableFileStoreWrite::Write(std::unique_ptr&& batch) { + if (write_ == nullptr) { + return Status::Invalid("format table write has been closed"); + } + return write_->Write(std::move(batch)); +} + +Status FormatTableFileStoreWrite::Compact(const std::map& partition, + int32_t bucket, bool full_compaction) { + return Status::NotImplemented( + "a format table cannot be compacted: it has no manifests to rewrite and no buckets to " + "compact within"); +} + +Result>> FormatTableFileStoreWrite::PrepareCommit( + bool wait_compaction, int64_t commit_identifier) { + if (write_ == nullptr) { + return Status::Invalid("format table write has been closed"); + } + // `wait_compaction` asks to wait rather than to do anything, and there is no compaction here + // to wait for, so it is honoured by returning at once rather than refused. + // + // A commit identifier is how a streaming write tells its attempts apart in a snapshot; a + // format table has no snapshots to record one in, so only a batch write fits here. + if (commit_identifier != BATCH_WRITE_COMMIT_IDENTIFIER) { + return Status::NotImplemented( + "a format table takes batch writes only: it has no snapshot to record a commit " + "identifier in"); + } + PAIMON_ASSIGN_OR_RAISE(std::vector messages, write_->PrepareCommit()); + std::vector> result; + result.reserve(messages.size()); + for (const FormatCommitMessage& message : messages) { + result.push_back(std::make_shared(message)); + } + return result; +} + +std::shared_ptr FormatTableFileStoreWrite::GetMetrics() const { + // Empty rather than null, so a caller merging metrics from several writers need not tell a + // table type that keeps none apart from one that does. What this write produced is on the + // commit messages it hands out. + return std::make_shared(); +} + +Status FormatTableFileStoreWrite::Close() { + // Closing drops the write rather than aborting it: a write that never prepared a commit clears + // what it staged from its own destructor, while one that has prepared has handed those files + // to a commit and must leave them alone. + write_.reset(); + return Status::OK(); +} + +} // namespace paimon diff --git a/src/paimon/core/table/format/format_table_file_store_write.h b/src/paimon/core/table/format/format_table_file_store_write.h new file mode 100644 index 00000000..551f1ed6 --- /dev/null +++ b/src/paimon/core/table/format/format_table_file_store_write.h @@ -0,0 +1,71 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#pragma once + +#include +#include +#include +#include +#include + +#include "paimon/defs.h" +#include "paimon/file_store_write.h" +#include "paimon/result.h" +#include "paimon/status.h" + +namespace paimon { + +class FormatTable; +class FormatTableWrite; +class MemoryPool; +class Metrics; + +/// Writes a format table through the `FileStoreWrite` interface, so that a caller holding a table +/// path writes it the way it writes any other table. Java Paimon does the same through +/// `FormatTable.newBatchWriteBuilder()`. +/// +/// A format table has no manifests and no snapshots, so the parts of `FileStoreWrite` that are +/// about them (compaction, and the commit identifier a streaming write carries) have nothing here +/// to act on and are refused rather than quietly ignored. +class FormatTableFileStoreWrite : public FileStoreWrite { + public: + static Result> Create( + const std::shared_ptr& table, const std::shared_ptr& pool); + + ~FormatTableFileStoreWrite() override; + + Status Write(std::unique_ptr&& batch) override; + + Status Compact(const std::map& partition, int32_t bucket, + bool full_compaction) override; + + Result>> PrepareCommit( + bool wait_compaction, int64_t commit_identifier) override; + + std::shared_ptr GetMetrics() const override; + + Status Close() override; + + private: + explicit FormatTableFileStoreWrite(std::unique_ptr&& write); + + std::unique_ptr write_; +}; + +} // namespace paimon diff --git a/src/paimon/core/table/format/format_table_loader.cpp b/src/paimon/core/table/format/format_table_loader.cpp new file mode 100644 index 00000000..fc541dff --- /dev/null +++ b/src/paimon/core/table/format/format_table_loader.cpp @@ -0,0 +1,80 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include "paimon/core/table/format/format_table_loader.h" + +#include +#include + +#include "paimon/catalog/identifier.h" +#include "paimon/common/utils/checked_cast.h" +#include "paimon/common/utils/path_util.h" +#include "paimon/core/options/table_type.h" +#include "paimon/core/schema/schema_manager.h" +#include "paimon/core/schema/table_schema.h" +#include "paimon/core/utils/branch_manager.h" +#include "paimon/schema/schema.h" +#include "paimon/table/format/format_table.h" + +namespace paimon { + +Result> FormatTableLoader::TryLoad( + const std::shared_ptr& file_system, const std::string& table_path, + const std::string& branch, const std::map& options, + const std::optional& specific_table_schema, const SchemaManager* schema_manager, + std::shared_ptr* table_schema_out) { + assert(table_schema_out != nullptr); + table_schema_out->reset(); + + std::shared_ptr table_schema; + if (branch == BranchManager::DEFAULT_MAIN_BRANCH && specific_table_schema) { + // Handing the schema over only saves reading it; it says nothing about where the table + // keeps its metadata. A context always names a table path and this library writes a + // table's metadata under it, so `schema` and `branch` below it stay metadata either way. + PAIMON_ASSIGN_OR_RAISE(table_schema, + TableSchema::CreateFromJson(specific_table_schema.value())); + } else { + // Through the caller's manager when it has one, so that the read warms the cache it goes + // on to use rather than a cache that dies with this call. + SchemaManager own_schema_manager(file_system, table_path, branch); + const SchemaManager& reader = + schema_manager != nullptr ? *schema_manager : own_schema_manager; + PAIMON_ASSIGN_OR_RAISE(std::optional> latest_schema, + reader.Latest()); + if (!latest_schema) { + return std::shared_ptr(); + } + table_schema = *latest_schema; + } + *table_schema_out = table_schema; + + // From the schema alone: a `type` given at the call must not decide what kind of table this + // is, not even for the length of that call. + PAIMON_ASSIGN_OR_RAISE(bool is_format_table, + TableTypeDefine::IsFormatTable(table_schema->Options())); + if (!is_format_table) { + return std::shared_ptr(); + } + // A context names a path, never a catalog identifier, so the table is known by the directory + // it sits in. + return FormatTable::Create(file_system, table_path, Identifier(PathUtil::GetName(table_path)), + checked_pointer_cast(table_schema), + /*location_carries_paimon_metadata=*/true, options); +} + +} // namespace paimon diff --git a/src/paimon/core/table/format/format_table_loader.h b/src/paimon/core/table/format/format_table_loader.h new file mode 100644 index 00000000..548b3dcb --- /dev/null +++ b/src/paimon/core/table/format/format_table_loader.h @@ -0,0 +1,72 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#pragma once + +#include +#include +#include +#include + +#include "paimon/result.h" + +namespace paimon { + +class FileSystem; +class FormatTable; +class SchemaManager; +class TableSchema; + +/// Recognises a format table behind a table path. +/// +/// The scan, read, write and commit entry points all take a path and have to answer the same +/// question before they can dispatch, so one place answers it and they cannot disagree. +class FormatTableLoader { + public: + FormatTableLoader() = delete; + ~FormatTableLoader() = delete; + + /// Loads `table_path` as a format table, or returns null when it is not one. + /// + /// Null is also the answer when there is no schema under the path at all: what to say about a + /// table that is not there is the managed path's to decide, and it says it in more detail. + /// + /// The table type is read from the schema, never from `options`: `type` is structural, and one + /// call's options must not decide what kind of table this is. + /// + /// @param branch Branch the schema is read from. Take it from the same place the managed path + /// in the same entry point takes it, or the two dispatch on different schemas. + /// @param options Options given at the call, which win over the ones the schema stored. + /// @param specific_table_schema A schema the caller already holds, as json. It saves reading + /// one from under the path and changes nothing else: the table still keeps its metadata + /// there. Used only on the main branch, as on the managed read path. + /// @param schema_manager The manager to read the schema through, or null to read it through + /// one of this call's own. A caller that goes on to use a `SchemaManager` itself must + /// pass that one: a manager caches the schemas it read, so reading through a second + /// one both costs an extra read and leaves the caller's cache cold. + /// @param table_schema_out Set to the schema this read, or to null when there is none under + /// the path. Every caller dispatches on that schema and then needs it again for the + /// managed table it turned out to be, so it is handed back rather than read twice. + static Result> TryLoad( + const std::shared_ptr& file_system, const std::string& table_path, + const std::string& branch, const std::map& options, + const std::optional& specific_table_schema, + const SchemaManager* schema_manager, std::shared_ptr* table_schema_out); +}; + +} // namespace paimon diff --git a/src/paimon/core/table/format/format_table_read.cpp b/src/paimon/core/table/format/format_table_read.cpp new file mode 100644 index 00000000..93383499 --- /dev/null +++ b/src/paimon/core/table/format/format_table_read.cpp @@ -0,0 +1,413 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include "paimon/core/table/format/format_table_read.h" + +#include +#include +#include +#include +#include +#include + +#include "arrow/api.h" +#include "arrow/c/bridge.h" +#include "arrow/c/helpers.h" +#include "fmt/format.h" +#include "paimon/common/data/binary_row.h" +#include "paimon/common/predicate/predicate_validator.h" +#include "paimon/common/reader/complete_row_kind_batch_reader.h" +#include "paimon/common/reader/concat_batch_reader.h" +#include "paimon/common/reader/data_file_reader_factory.h" +#include "paimon/common/reader/predicate_batch_reader.h" +#include "paimon/common/types/data_field.h" +#include "paimon/common/utils/arrow/status_utils.h" +#include "paimon/common/utils/binary_row_partition_computer.h" +#include "paimon/common/utils/scope_guard.h" +#include "paimon/core/core_options.h" +#include "paimon/core/io/field_mapping_reader.h" +#include "paimon/core/table/format/format_data_split.h" +#include "paimon/core/table/format/format_path_validation.h" +#include "paimon/core/table/format/lazy_concat_batch_reader.h" +#include "paimon/core/utils/field_mapping.h" +#include "paimon/format/file_format.h" +#include "paimon/format/file_format_factory.h" +#include "paimon/fs/file_system.h" +#include "paimon/predicate/predicate.h" + +namespace paimon { + +/// Readers, outermost first: CompleteRowKindBatchReader -> (PredicateBatchReader) +/// -> LazyConcatBatchReader across the split's files -> FieldMappingReader +/// -> (DelegatingPrefetchReader) -> (PrefetchFileBatchReader) -> FormatReader +/// +/// The same shape the managed table path builds, minus what a format table has none of: no +/// deletion vectors, no bitmap index, no row-tracking fields and no shredding. The last three +/// readers are built by `DataFileReaderFactory`, which is where the two paths meet. +class FormatTableRead::Impl { + public: + std::shared_ptr table; + /// Columns the reader returns, in the order it returns them. + std::shared_ptr read_schema; + /// The whole table schema: the mapping below splits the partition columns out of it. + std::shared_ptr data_schema; + /// Splits the read schema into file columns and partition columns and rewrites the predicate + /// against the file's own fields. The same builder the managed table path uses. + std::shared_ptr field_mapping_builder; + /// Turns a split's partition values into the `BinaryRow` a `FieldMappingReader` fills its + /// partition columns from. Null when the table is not partitioned. + std::shared_ptr partition_computer; + /// The predicate the returned reader applies exactly, or null when the caller filters itself. + std::shared_ptr filter_predicate; + std::shared_ptr pool; + /// Runs the reads a prefetching reader issues ahead of the batches being asked for. Null when + /// nothing asked for prefetch. + std::shared_ptr executor; + std::string format_identifier; + /// What a file is opened with. The same struct the managed table path fills in, read by the + /// same component. + DataFileReadOptions read_options; +}; + +FormatTableRead::FormatTableRead(std::unique_ptr impl, + const std::shared_ptr& pool) + : TableRead(pool), impl_(std::move(impl)) {} + +FormatTableRead::~FormatTableRead() = default; + +Result> FormatTableRead::Create( + const std::shared_ptr& table, + const std::optional>& projection, + const std::shared_ptr& pool, const std::shared_ptr& predicate, + bool enable_predicate_filter) { + return CreateInternal(table, projection, pool, predicate, enable_predicate_filter, + /*read_context=*/nullptr); +} + +Result> FormatTableRead::Create( + const std::shared_ptr& table, const std::shared_ptr& read_context) { + if (table == nullptr) { + return Status::Invalid("format table read requires a table"); + } + if (read_context == nullptr) { + return Status::Invalid("format table read requires a read context"); + } + if (read_context->GetRealtimeContext() != nullptr) { + return Status::NotImplemented( + "a format table has no real-time store to union with what is on disk"); + } + // A projected read schema can rename a column, prune a nested one and give it metadata of its + // own, while a format table's projection is a list of top-level names, so it is refused rather + // than read as if it had never been given. + if (read_context->GetReadSchema() != nullptr) { + return Status::NotImplemented( + "a format table read does not take a projected read schema; name the columns to read " + "instead"); + } + + std::optional> projection; + if (!read_context->GetReadFieldNames().empty()) { + projection = read_context->GetReadFieldNames(); + } else if (!read_context->GetReadFieldIds().empty()) { + // Resolved against the table's own schema, which is the only thing that knows the ids: a + // file another engine wrote carries none. + PAIMON_ASSIGN_OR_RAISE(std::unique_ptr<::ArrowSchema> c_schema, table->GetArrowSchema()); + PAIMON_ASSIGN_OR_RAISE_FROM_ARROW(std::shared_ptr table_schema, + arrow::ImportSchema(c_schema.get())); + PAIMON_ASSIGN_OR_RAISE(std::vector fields, + DataField::ConvertArrowSchemaToDataFields(table_schema)); + std::map name_by_id; + for (const DataField& field : fields) { + name_by_id.emplace(field.Id(), field.Name()); + } + std::vector names; + names.reserve(read_context->GetReadFieldIds().size()); + for (int32_t field_id : read_context->GetReadFieldIds()) { + auto iter = name_by_id.find(field_id); + if (iter == name_by_id.end()) { + return Status::Invalid(fmt::format("field id {} is not a column of table {}", + field_id, table->FullName())); + } + names.push_back(iter->second); + } + projection = std::move(names); + } + + return CreateInternal(table, projection, read_context->GetMemoryPool(), + read_context->GetPredicate(), read_context->EnablePredicateFilter(), + read_context); +} + +Result> FormatTableRead::CreateInternal( + const std::shared_ptr& table, + const std::optional>& projection, + const std::shared_ptr& pool, const std::shared_ptr& predicate, + bool enable_predicate_filter, const std::shared_ptr& read_context) { + if (table == nullptr) { + return Status::Invalid("format table read requires a table"); + } + std::shared_ptr memory_pool = pool != nullptr ? pool : GetDefaultPool(); + + PAIMON_ASSIGN_OR_RAISE(std::unique_ptr<::ArrowSchema> c_schema, table->GetArrowSchema()); + PAIMON_ASSIGN_OR_RAISE_FROM_ARROW(std::shared_ptr table_schema, + arrow::ImportSchema(c_schema.get())); + + const std::vector& partition_keys = table->PartitionKeys(); + auto is_partition_key = [&partition_keys](const std::string& name) { + return std::find(partition_keys.begin(), partition_keys.end(), name) != + partition_keys.end(); + }; + + arrow::FieldVector read_fields; + if (projection) { + read_fields.reserve(projection->size()); + std::set projected; + for (const std::string& name : *projection) { + // A read column is looked up by name, so twice has no meaning to act on. + if (!projected.insert(name).second) { + return Status::Invalid(fmt::format( + "column '{}' appears more than once in the projection, which paimon-cpp does " + "not allow", + name)); + } + std::shared_ptr field = table_schema->GetFieldByName(name); + if (field == nullptr) { + return Status::Invalid( + fmt::format("field '{}' is not a column of table {}", name, table->FullName())); + } + read_fields.push_back(std::move(field)); + } + } else { + read_fields = table_schema->fields(); + } + if (read_fields.empty()) { + return Status::Invalid("format table read requires at least one column to read"); + } + + auto impl = std::make_unique(); + impl->table = table; + impl->read_schema = arrow::schema(read_fields); + impl->pool = memory_pool; + impl->format_identifier = FormatTable::FormatToString(table->GetFormat()); + // The whole schema, as the managed table path hands it over: the mapping splits the + // partition columns out itself and asks the file only for what is left. + impl->data_schema = table_schema; + + const bool has_non_partition_column = + std::any_of(table_schema->fields().begin(), table_schema->fields().end(), + [&is_partition_key](const std::shared_ptr& field) { + return !is_partition_key(field->name()); + }); + if (!has_non_partition_column) { + return Status::Invalid( + fmt::format("format table {} has no non-partition column, so its files hold nothing to " + "read", + table->FullName())); + } + + if (predicate != nullptr) { + // The same rules `InternalReadContext` applies to a managed table's predicate. The field + // index is not among them: everything downstream resolves a field by name. + PAIMON_RETURN_NOT_OK(PredicateValidator::ValidatePredicateWithSchema( + *impl->read_schema, predicate, /*validate_field_idx=*/false)); + PAIMON_RETURN_NOT_OK(PredicateValidator::ValidatePredicateWithLiterals(predicate)); + if (enable_predicate_filter) { + impl->filter_predicate = predicate; + } + } + + // The builder also hands the file reader only the conjuncts naming columns the file holds. + PAIMON_ASSIGN_OR_RAISE( + std::unique_ptr field_mapping_builder, + FieldMappingBuilder::Create(impl->read_schema, partition_keys, predicate)); + impl->field_mapping_builder = std::move(field_mapping_builder); + + PAIMON_ASSIGN_OR_RAISE(CoreOptions core_options, + CoreOptions::FromMap(table->Options(), table->GetFileSystem())); + impl->read_options.read_batch_size = core_options.GetReadBatchSize(); + impl->read_options.adaptive_prefetch_strategy = core_options.EnableAdaptivePrefetchStrategy(); + if (read_context != nullptr) { + // Straight from the context, as the managed table path takes them. Without a context + // nobody asked for any of this, so a file is opened plainly. + impl->read_options.cache = read_context->GetCache(); + impl->read_options.prefetch_enabled = read_context->EnablePrefetch(); + impl->read_options.prefetch_max_parallel_num = read_context->GetPrefetchMaxParallelNum(); + impl->read_options.prefetch_batch_count = read_context->GetPrefetchBatchCount(); + impl->read_options.read_ahead_cache_enabled = read_context->ReadAheadCacheEnabled(); + impl->read_options.cache_config = read_context->GetCacheConfig(); + impl->executor = read_context->GetExecutor(); + } + if (!partition_keys.empty()) { + PAIMON_ASSIGN_OR_RAISE(std::unique_ptr partition_computer, + BinaryRowPartitionComputer::Create( + partition_keys, table_schema, table->PartitionDefaultName(), + core_options.LegacyPartitionNameEnabled(), memory_pool)); + impl->partition_computer = std::move(partition_computer); + } + + return std::unique_ptr(new FormatTableRead(std::move(impl), memory_pool)); +} + +Result> FormatTableRead::CreateSplitReader( + const std::shared_ptr& split) { + auto format_split = std::dynamic_pointer_cast(split); + if (format_split == nullptr) { + return Status::Invalid("format table read only accepts a FormatDataSplit"); + } + + // `CreateReader()` takes a `Split` the caller held on to, which may have been planned from + // another table or before these files moved, so whether a file belongs to this table is asked + // here rather than taken on trust. + PAIMON_RETURN_NOT_OK(FormatPathValidation::ValidatePartitionKeys( + impl_->table, format_split->partition, "split")); + for (const FormatDataSplit::FileMeta& file : format_split->files) { + PAIMON_RETURN_NOT_OK(FormatPathValidation::ValidatePathUnderLocation( + file.file_path, impl_->table->Location(), "split")); + // A split mixing partitions would read rows back under values they never had. + PAIMON_RETURN_NOT_OK(FormatPathValidation::ValidateFileInPartition( + impl_->table, file.file_path, format_split->partition, "split")); + PAIMON_RETURN_NOT_OK( + FormatPathValidation::ValidateFileIsVisible(impl_->table, file.file_path, "split")); + if (file.file_size < 0) { + return Status::Invalid(fmt::format("split gives {} a negative size", file.file_path)); + } + } + + // The partition values in the shape a `FieldMappingReader` reads them from; a directory named + // after the default partition name reads back as null. + BinaryRow partition = BinaryRow::EmptyRow(); + if (impl_->partition_computer != nullptr) { + PAIMON_ASSIGN_OR_RAISE(partition, + impl_->partition_computer->ToBinaryRow(format_split->partition)); + } + + // One reader builder serves the whole split, built by the same component the managed table + // path uses, so a format table's file is read with the cache and the read hints any other + // data file is. + PAIMON_ASSIGN_OR_RAISE(std::unique_ptr builder, + DataFileReaderFactory::CreateReaderBuilder( + impl_->format_identifier, impl_->table->Options(), + /*extra_format_options=*/{}, impl_->read_options, impl_->pool)); + std::shared_ptr reader_builder(std::move(builder)); + + // Captured by value, so a file's reader outlives this `FormatTableRead`. + std::shared_ptr table = impl_->table; + std::shared_ptr data_schema = impl_->data_schema; + std::shared_ptr field_mapping_builder = impl_->field_mapping_builder; + std::shared_ptr pool = impl_->pool; + std::shared_ptr executor = impl_->executor; + std::string format_identifier = impl_->format_identifier; + DataFileReadOptions read_options = impl_->read_options; + + // Each file is named alongside its factory, so every failure says which file it was. + std::vector sources; + sources.reserve(format_split->files.size()); + for (const FormatDataSplit::FileMeta& file : format_split->files) { + LazyConcatBatchReader::Source source; + source.name = file.file_path; + source.open = [table, reader_builder, data_schema, field_mapping_builder, partition, pool, + executor, format_identifier, read_options, + file]() -> Result> { + // The same mapping for every file; built per file only because `FieldMappingReader` + // takes ownership of it. + PAIMON_ASSIGN_OR_RAISE(std::unique_ptr field_mapping, + field_mapping_builder->CreateFieldMapping(data_schema)); + std::shared_ptr file_read_schema = + DataField::ConvertDataFieldsToArrowSchema( + field_mapping->non_partition_info.non_partition_data_schema); + std::shared_ptr pushdown_predicate = + field_mapping->non_partition_info.non_partition_filter; + + // The split's size is whatever the caller gave it and `Open` trusts what it is handed: + // a stale length would truncate an object-store read or send it past the end, so the + // file system is asked for the real one. + PAIMON_ASSIGN_OR_RAISE(FileStatus status, + table->GetFileSystem()->GetFileStatus(file.file_path)); + if (status.IsDir()) { + return Status::Invalid("the split names a directory, not a data file"); + } + if (file.file_size != status.GetLen()) { + return Status::Invalid(fmt::format( + "the split says it is {} bytes but it is {}; the plan was made against a " + "different version of the file", + file.file_size, status.GetLen())); + } + // Opened through the same component the managed table path opens a data file with, + // so prefetch and the read-ahead cache apply here too. The size is the one the file + // system just reported, not the one the split claimed. + PAIMON_ASSIGN_OR_RAISE( + std::unique_ptr file_reader, + DataFileReaderFactory::Open(format_identifier, file.file_path, status.GetLen(), + reader_builder.get(), read_options, + table->GetFileSystem(), executor, pool)); + + ::ArrowSchema c_read_schema; + ArrowSchemaMarkReleased(&c_read_schema); + ScopeGuard read_schema_guard( + [&c_read_schema]() { ArrowSchemaRelease(&c_read_schema); }); + PAIMON_RETURN_NOT_OK_FROM_ARROW(arrow::ExportSchema(*file_read_schema, &c_read_schema)); + PAIMON_RETURN_NOT_OK(file_reader->SetReadSchema(&c_read_schema, pushdown_predicate, + /*selection_bitmap=*/std::nullopt)); + + PAIMON_ASSIGN_OR_RAISE(std::unique_ptr reader, + FieldMappingReader::Create( + field_mapping_builder->GetReadFieldCount(), + std::move(file_reader), partition, std::move(field_mapping), + /*skip_map_selected_keys_filter_field_ids=*/{}, pool)); + return std::unique_ptr(std::move(reader)); + }; + sources.push_back(std::move(source)); + } + + return std::make_unique(std::move(sources), impl_->pool); +} + +Result> FormatTableRead::ApplyFilterAndRowKind( + std::unique_ptr&& reader) { + std::unique_ptr result = std::move(reader); + if (impl_->filter_predicate != nullptr) { + PAIMON_ASSIGN_OR_RAISE( + std::unique_ptr filtered, + PredicateBatchReader::Create(std::move(result), impl_->filter_predicate, impl_->pool)); + result = std::move(filtered); + } + // Every row is an insert, but `BatchReader::NextBatch()` still promises the leading + // `_VALUE_KIND` field: an engine reading by field index would find its columns shifted. + return std::make_unique(std::move(result), impl_->pool); +} + +Result> FormatTableRead::CreateReader( + const std::shared_ptr& split) { + PAIMON_ASSIGN_OR_RAISE(std::unique_ptr reader, CreateSplitReader(split)); + return ApplyFilterAndRowKind(std::move(reader)); +} + +Result> FormatTableRead::CreateReader( + const std::vector>& splits) { + std::vector> split_readers; + split_readers.reserve(splits.size()); + for (const std::shared_ptr& split : splits) { + PAIMON_ASSIGN_OR_RAISE(std::unique_ptr reader, CreateSplitReader(split)); + split_readers.push_back(std::move(reader)); + } + std::unique_ptr reader = + std::make_unique(std::move(split_readers), impl_->pool); + return ApplyFilterAndRowKind(std::move(reader)); +} + +} // namespace paimon diff --git a/src/paimon/core/table/format/format_table_read.h b/src/paimon/core/table/format/format_table_read.h new file mode 100644 index 00000000..69170e85 --- /dev/null +++ b/src/paimon/core/table/format/format_table_read.h @@ -0,0 +1,120 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#pragma once + +#include +#include +#include +#include + +#include "paimon/memory/memory_pool.h" +#include "paimon/read_context.h" +#include "paimon/reader/batch_reader.h" +#include "paimon/result.h" +#include "paimon/table/format/format_table.h" +#include "paimon/table/source/split.h" +#include "paimon/table/source/table_read.h" + +namespace paimon { + +class Predicate; + +/// Reads the splits a `FormatTableScan` produced. +/// +/// The batches carry the leading `_VALUE_KIND` field every `BatchReader` promises, filled with +/// inserts: a directory of plain data files records no row kind and every row in it is an insert. +/// Partition columns are rebuilt from the split's partition values, since the data files do not +/// carry them. +/// +/// A batch borrows memory from the reader that produced it, so every batch must be released before +/// that reader is destroyed. +/// +/// Building a reader leaves the read as it was, so one may be shared between threads; the +/// `BatchReader`s it hands out may not be, as `TableRead` says. +/// +/// `CreateCountReader()` is not implemented and falls through to `TableRead`'s default, which +/// refuses: counting a format table's rows means reading them. +class FormatTableRead : public TableRead { + public: + /// @param table Table the splits belong to. + /// @param projection Names of the columns to read, in the order they should appear. When + /// absent, every column of the table is read. A column named twice is rejected. + /// @param pool Memory pool the batches are allocated from. + /// @param predicate Rows the caller is interested in, as a filter over the columns being read. + /// It is pushed into the file readers so the format skips what its own statistics let + /// it skip; on its own that is a best effort and rows the predicate rejects can still + /// come back. As on the managed table path, every field it names must be one the + /// projection keeps, of the type the conjunct declares, with literals of that same type + /// and none of them null. Field indexes are ignored: a field is resolved by name. + /// @param enable_predicate_filter Whether the returned reader applies `predicate` exactly to + /// the rows it returns. Off by default, as everywhere else in paimon-cpp. + static Result> Create( + const std::shared_ptr& table, + const std::optional>& projection, + const std::shared_ptr& pool, const std::shared_ptr& predicate, + bool enable_predicate_filter); + + /// Reads what a `ReadContext` asks for, which is how `TableRead::Create()` reaches a format + /// table. The columns to read, the predicate and whether to apply it exactly, the memory pool + /// and executor, and what a file is opened with (prefetch, the read-ahead cache and the block + /// cache) all come from the context. The last three go through the same component the managed + /// table path opens its files with, so a format table's file is read the way any other data + /// file is. + /// + /// A setting a format table cannot honour is refused by name rather than dropped: a projected + /// read schema, and a real-time context. + static Result> Create( + const std::shared_ptr& table, + const std::shared_ptr& read_context); + + ~FormatTableRead() override; + + /// Creates a reader over one split's files, read in the split's order. + Result> CreateReader(const std::shared_ptr& split) override; + + /// Creates a reader over several splits, read in the given order. + Result> CreateReader( + const std::vector>& splits) override; + + class Impl; + + private: + explicit FormatTableRead(std::unique_ptr impl, const std::shared_ptr& pool); + + /// Shared body of both `Create()` overloads. A null `read_context` reads with neither prefetch + /// nor a cache. + static Result> CreateInternal( + const std::shared_ptr& table, + const std::optional>& projection, + const std::shared_ptr& pool, const std::shared_ptr& predicate, + bool enable_predicate_filter, const std::shared_ptr& read_context); + + /// Builds the reader over one split's files, without the predicate filter or the row kinds. + Result> CreateSplitReader(const std::shared_ptr& split); + + /// Wraps a reader in the exact predicate filter when one was asked for, and then in the + /// `_VALUE_KIND` field every `BatchReader` promises. The filter runs underneath, on the + /// table's own columns, as it does on the managed table path. + Result> ApplyFilterAndRowKind( + std::unique_ptr&& reader); + + std::unique_ptr impl_; +}; + +} // namespace paimon diff --git a/src/paimon/core/table/format/format_table_scan.cpp b/src/paimon/core/table/format/format_table_scan.cpp new file mode 100644 index 00000000..bb80a18c --- /dev/null +++ b/src/paimon/core/table/format/format_table_scan.cpp @@ -0,0 +1,236 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include "paimon/core/table/format/format_table_scan.h" + +#include +#include +#include + +#include "fmt/format.h" +#include "fmt/ranges.h" +#include "paimon/common/utils/bin_packing.h" +#include "paimon/common/utils/path_util.h" +#include "paimon/core/core_options.h" +#include "paimon/core/table/format/format_data_split.h" +#include "paimon/core/table/format/format_file_listing.h" +#include "paimon/core/table/format/format_path_validation.h" +#include "paimon/core/table/source/plan_impl.h" +#include "paimon/core/utils/partition_path_utils.h" +#include "paimon/fs/file_system.h" +#include "paimon/logging.h" + +namespace paimon { + +namespace { +Logger* ScanLogger() { + static std::unique_ptr logger = Logger::GetLogger("FormatTableScan"); + return logger.get(); +} +} // namespace + +FormatTableScan::FormatTableScan(const std::shared_ptr& table, + const std::map& partition_filter, + const std::optional& limit, int64_t target_split_size, + int64_t open_file_cost) + : table_(table), + partition_filter_(partition_filter), + limit_(limit), + target_split_size_(target_split_size), + open_file_cost_(open_file_cost) {} + +FormatTableScan::~FormatTableScan() = default; + +Result> FormatTableScan::Create( + const std::shared_ptr& table, + const std::map& partition_filter, + const std::optional& limit) { + if (table == nullptr) { + return Status::Invalid("format table scan requires a table"); + } + const std::vector& partition_keys = table->PartitionKeys(); + for (const auto& filter : partition_filter) { + const std::string& key = filter.first; + if (std::find(partition_keys.begin(), partition_keys.end(), key) == partition_keys.end()) { + return Status::Invalid( + fmt::format("partition filter field '{}' is not a partition key of table {}", key, + table->FullName())); + } + } + // Through `CoreOptions`, so `"128 mb"` means what it does elsewhere and a default lives in + // one place. + PAIMON_ASSIGN_OR_RAISE(CoreOptions core_options, + CoreOptions::FromMap(table->Options(), table->GetFileSystem())); + return std::unique_ptr( + new FormatTableScan(table, partition_filter, limit, core_options.GetSourceSplitTargetSize(), + core_options.GetSourceSplitOpenFileCost())); +} + +Result> FormatTableScan::FindPartitions() const { + const std::vector& partition_keys = table_->PartitionKeys(); + // Partitions are discovered by listing one directory level at a time, so the filter applied + // and the number of partitions that survived it are the only way to explain an empty plan. + PAIMON_LOG_DEBUG(ScanLogger(), "Finding partitions for format table %s, partition filter: %s", + table_->FullName().c_str(), fmt::format("{}", partition_filter_).c_str()); + std::shared_ptr file_system = table_->GetFileSystem(); + + const bool only_value = table_->PartitionOnlyValueInPath(); + // The one hidden name that is table content: a null partition's directory in the value-only + // layout. + const std::string& default_partition_name = table_->PartitionDefaultName(); + + // One partition level at a time, keeping each partition paired with its directory. + std::vector level = { + {std::map(), table_->Location()}}; + bool at_table_location = true; + for (const std::string& partition_key : partition_keys) { + std::vector next; + for (const auto& [partition, directory] : level) { + std::vector children; + Status status = file_system->ListDir(directory, &children); + if (status.IsNotExist()) { + // Gone since it was listed, or a table directory not created yet; either way + // the rest of the listing stands. + continue; + } + PAIMON_RETURN_NOT_OK(status); + for (const BasicFileStatus& child : children) { + if (!child.IsDir()) { + continue; + } + std::string name = PathUtil::GetName(child.GetPath()); + if (PartitionPathUtils::IsHiddenName(name) && + !(only_value && name == default_partition_name)) { + continue; + } + if (at_table_location && table_->LocationCarriesPaimonMetadata() && + FormatFileListing::IsReservedDirectory(name)) { + continue; + } + std::string value; + if (only_value) { + // Nothing but the level says which key a directory belongs to, so every + // directory here is one. + value = PartitionPathUtils::UnescapePathName(name); + } else { + std::optional> key_value = + PartitionPathUtils::ExtractPartitionKeyValue(name); + if (!key_value || key_value->first != partition_key) { + // Something else lives here: another layout, or a nested table. + continue; + } + value = std::move(key_value->second); + } + auto filter_iter = partition_filter_.find(partition_key); + if (filter_iter != partition_filter_.end() && filter_iter->second != value) { + continue; + } + std::map child_partition = partition; + child_partition[partition_key] = std::move(value); + next.emplace_back(std::move(child_partition), child.GetPath()); + } + } + level = std::move(next); + at_table_location = false; + } + PAIMON_LOG_DEBUG(ScanLogger(), "Found %zu partitions of format table %s", level.size(), + table_->FullName().c_str()); + return level; +} + +Result>> FormatTableScan::CreateSplits( + const std::string& directory, const std::map& partition) const { + // `directory` is a complete partition, or the table itself when unpartitioned, so no partition + // directory is left below it and `partition_levels` stays 0. + FormatDataFileListingOptions listing; + // Only right at the table's location are `schema` and `branch` its metadata. + PAIMON_ASSIGN_OR_RAISE(bool at_location, + FormatPathValidation::IsTableLocation(table_, directory)); + listing.skip_reserved_directories = at_location && table_->LocationCarriesPaimonMetadata(); + std::vector files; + PAIMON_RETURN_NOT_OK( + FormatFileListing::ListDataFiles(table_->GetFileSystem(), directory, listing, &files)); + std::vector> splits; + if (files.empty()) { + return splits; + } + // The file system promises no listing order, and a scan's output should be reproducible. + std::sort(files.begin(), files.end(), + [](const FormatDataSplit::FileMeta& left, const FormatDataSplit::FileMeta& right) { + return left.file_path < right.file_path; + }); + // A file is never cut up: parquet and orc each record where their own row groups and stripes + // begin. An entry costs at least the open-file cost, so a split cannot gather so many small + // files that opening them outweighs reading them. + const int64_t open_file_cost = open_file_cost_; + std::vector> bins = + BinPacking::PackForOrdered( + std::move(files), + [open_file_cost](const FormatDataSplit::FileMeta& file) { + return std::max(file.file_size, open_file_cost); + }, + target_split_size_); + splits.reserve(bins.size()); + for (const std::vector& bin : bins) { + splits.push_back(std::make_shared(bin, partition)); + } + return splits; +} + +Result> FormatTableScan::CreatePlan() { + std::vector> splits; + // A non-positive limit asks for no rows at all, so there is nothing to read. + if (limit_ && *limit_ <= 0) { + return std::make_shared(std::nullopt, splits); + } + + if (table_->PartitionKeys().empty()) { + PAIMON_ASSIGN_OR_RAISE(splits, CreateSplits(table_->Location(), {})); + return std::make_shared(std::nullopt, splits); + } + + PAIMON_ASSIGN_OR_RAISE(std::vector partitions, FindPartitions()); + // A stable order, like the files within a partition. + std::sort(partitions.begin(), partitions.end(), + [](const auto& left, const auto& right) { return left.second < right.second; }); + for (const auto& [partition, directory] : partitions) { + PAIMON_ASSIGN_OR_RAISE(std::vector> partition_splits, + CreateSplits(directory, partition)); + splits.insert(splits.end(), std::make_move_iterator(partition_splits.begin()), + std::make_move_iterator(partition_splits.end())); + } + return std::make_shared(std::nullopt, splits); +} + +Result>> FormatTableScan::ListPartitions() const { + std::vector> result; + if (table_->PartitionKeys().empty()) { + return result; + } + PAIMON_ASSIGN_OR_RAISE(std::vector partitions, FindPartitions()); + // A stable order, like a plan's splits. + std::sort(partitions.begin(), partitions.end(), + [](const auto& left, const auto& right) { return left.second < right.second; }); + result.reserve(partitions.size()); + for (PartitionAndPath& partition_and_path : partitions) { + result.push_back(std::move(partition_and_path.first)); + } + return result; +} + +} // namespace paimon diff --git a/src/paimon/core/table/format/format_table_scan.h b/src/paimon/core/table/format/format_table_scan.h new file mode 100644 index 00000000..37a33607 --- /dev/null +++ b/src/paimon/core/table/format/format_table_scan.h @@ -0,0 +1,107 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#pragma once + +#include +#include +#include +#include +#include +#include + +#include "paimon/result.h" +#include "paimon/table/format/format_table.h" +#include "paimon/table/source/plan.h" +#include "paimon/table/source/table_scan.h" + +namespace paimon { + +/// Plans a read of a format table by listing its directories. +/// +/// Partitions are discovered from the directory layout: every `key=value` directory that matches +/// the table's partition keys, nested in the order the keys are declared, or, when the table sets +/// `format-table.partition-path-only-value`, every directory at that level, whose bare name is the +/// value. Data files are then collected from a partition directory and everything below it. +/// +/// Names starting with `_` or `.` are skipped and not descended into, since that is how an engine +/// marks output that is not committed table data. The one exception is the directory standing for +/// a null partition value in the value-only layout, named by `partition.default-name`. +/// +/// A partition's files are packed into splits of about `source.split.target-size`, counting each +/// file as at least `source.split.open-file-cost`. A file larger than that target is still a split +/// of its own: see `FormatDataSplit` for why one file is never shared between readers. +/// +/// Planning leaves the scan as it was, so one may be shared between threads; what it plans is +/// whatever the directories held when it looked. +class FormatTableScan : public TableScan { + public: + /// A partition's values paired with the directory they were read from. + using PartitionAndPath = std::pair, std::string>; + + /// @param table Table to scan. + /// @param partition_filter Partition values to keep, keyed by partition field name. A key that + /// is absent from the map is unconstrained, so a filter may name only some of the + /// partition keys. An empty map keeps every partition. + /// @param limit Upper bound on the rows the caller will read. Splits are dropped once the + /// remaining ones cannot be needed, but a format table records no row counts, so a + /// positive limit cannot drop anything and the caller still has to stop reading itself. + static Result> Create( + const std::shared_ptr& table, + const std::map& partition_filter, + const std::optional& limit); + + ~FormatTableScan() override; + + /// Plans the read: the splits of every partition that passes the filter, in a stable order. + /// + /// A missing table directory is answered differently by the two shapes of table. An + /// unpartitioned table has only its location to list, so a location that is not there fails, + /// which is the only way to tell a wrong location from a table with no rows. A partitioned + /// table discovers its partitions by descending, so an absent directory simply has no + /// partitions below it and the plan comes back empty. + Result> CreatePlan() override; + + /// Lists the partitions the table's directory layout holds, whether or not they hold data, + /// in a stable order. Every partition key gets a value; a directory named + /// `partition.default-name` reads back as that name, standing for a null partition value. + Result>> ListPartitions() const override; + + private: + FormatTableScan(const std::shared_ptr& table, + const std::map& partition_filter, + const std::optional& limit, int64_t target_split_size, + int64_t open_file_cost); + + /// Lists partition directories under the table location that pass the partition filter, paired + /// with the partition values their names spell out. + Result> FindPartitions() const; + + /// Collects the data files under `directory` and everything below it and packs them into + /// splits of about the target size. Empty when there is no data file. + Result>> CreateSplits( + const std::string& directory, const std::map& partition) const; + + std::shared_ptr table_; + std::map partition_filter_; + std::optional limit_; + int64_t target_split_size_; + int64_t open_file_cost_; +}; + +} // namespace paimon diff --git a/src/paimon/core/table/format/format_table_test.cpp b/src/paimon/core/table/format/format_table_test.cpp new file mode 100644 index 00000000..19b22b3b --- /dev/null +++ b/src/paimon/core/table/format/format_table_test.cpp @@ -0,0 +1,3176 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include "paimon/table/format/format_table.h" + +#include +#include +#include +#include +#include +#include +#include +#include + +#include "arrow/api.h" +#include "arrow/c/bridge.h" +#include "gtest/gtest.h" +#include "paimon/cache/cache.h" +#include "paimon/commit_context.h" +#include "paimon/commit_message.h" +#include "paimon/common/table/special_fields.h" +#include "paimon/common/types/row_kind.h" +#include "paimon/common/utils/arrow/status_utils.h" +#include "paimon/common/utils/checked_cast.h" +#include "paimon/common/utils/path_util.h" +#include "paimon/common/utils/string_utils.h" +#include "paimon/core/schema/schema_manager.h" +#include "paimon/core/schema/table_schema.h" +#include "paimon/core/table/format/format_commit_message.h" +#include "paimon/core/table/format/format_data_split.h" +#include "paimon/core/table/format/format_table_commit.h" +#include "paimon/core/table/format/format_table_read.h" +#include "paimon/core/table/format/format_table_scan.h" +#include "paimon/core/table/format/format_table_write.h" +#include "paimon/defs.h" +#include "paimon/file_store_commit.h" +#include "paimon/file_store_write.h" +#include "paimon/fs/file_system.h" +#include "paimon/memory/memory_pool.h" +#include "paimon/predicate/literal.h" +#include "paimon/predicate/predicate_builder.h" +#include "paimon/read_context.h" +#include "paimon/record_batch.h" +#include "paimon/scan_context.h" +#include "paimon/status.h" +#include "paimon/table/source/split.h" +#include "paimon/table/source/table_read.h" +#include "paimon/table/source/table_scan.h" +#include "paimon/testing/utils/testharness.h" +#include "paimon/write_context.h" + +namespace paimon::test { + +namespace { + +/// What `ListPartitions()` returns. Aliased because a macro argument cannot hold the comma in +/// `std::map`: the preprocessor would read it as two arguments. +using PartitionList = std::vector>; + +std::shared_ptr MakeSchema() { + return arrow::schema({arrow::field("id", arrow::int32()), arrow::field("name", arrow::utf8()), + arrow::field("dt", arrow::utf8())}); +} + +/// Wraps a file system and writes down what a write did to each path, in order. Only the order +/// gives away a temp file deleted while its stream is still open, which on a store that flushes +/// from the destructor lands the write after the delete. +class CallOrderFileSystem : public FileSystem { + public: + explicit CallOrderFileSystem(const std::shared_ptr& delegate) + : delegate_(delegate), calls_(std::make_shared>()) {} + + /// What happened, as " " in the order it happened. + const std::vector& Calls() const { + return *calls_; + } + + using FileSystem::Open; + + Result> Open(const std::string& path) const override { + return delegate_->Open(path); + } + Result> Create(const std::string& path, + bool overwrite) const override { + PAIMON_ASSIGN_OR_RAISE(std::unique_ptr out, + delegate_->Create(path, overwrite)); + calls_->push_back("create " + path); + return std::unique_ptr( + new RecordingOutputStream(std::move(out), path, calls_)); + } + Status Mkdirs(const std::string& path) const override { + return delegate_->Mkdirs(path); + } + Status Rename(const std::string& src, const std::string& dst) const override { + return delegate_->Rename(src, dst); + } + Status Delete(const std::string& path, bool recursive = true) const override { + calls_->push_back("delete " + path); + return delegate_->Delete(path, recursive); + } + Result GetFileStatus(const std::string& path) const override { + return delegate_->GetFileStatus(path); + } + Status ListDir(const std::string& directory, + std::vector* status_list) const override { + return delegate_->ListDir(directory, status_list); + } + Status ListFileStatus(const std::string& path, + std::vector* status_list) const override { + return delegate_->ListFileStatus(path, status_list); + } + Result Exists(const std::string& path) const override { + return delegate_->Exists(path); + } + + private: + /// Records its own close, so a stream still open when its file was deleted can be told apart + /// from one that was closed first. + class RecordingOutputStream : public OutputStream { + public: + RecordingOutputStream(std::unique_ptr delegate, const std::string& path, + const std::shared_ptr>& calls) + : delegate_(std::move(delegate)), path_(path), calls_(calls) {} + + Result Write(const char* buffer, int64_t size) override { + return delegate_->Write(buffer, size); + } + Status Flush() override { + return delegate_->Flush(); + } + Result GetPos() const override { + return delegate_->GetPos(); + } + Result GetUri() const override { + return delegate_->GetUri(); + } + Status Close() override { + calls_->push_back("close " + path_); + return delegate_->Close(); + } + + private: + std::unique_ptr delegate_; + std::string path_; + std::shared_ptr> calls_; + }; + + std::shared_ptr delegate_; + /// Shared with every stream this hands out, so one list holds the whole sequence. + std::shared_ptr> calls_; +}; + +/// The one call a `FailingWriteFileSystem`'s streams refuse. +enum class FailingStreamCall { kGetPos, kFlush, kClose }; + +/// Wraps a file system and hands out streams that fail one call, so that a write can be stopped +/// where a real store would stop it: after the writer is finished and while the file it produced +/// is still hidden. +class FailingWriteFileSystem : public FileSystem { + public: + FailingWriteFileSystem(const std::shared_ptr& delegate, FailingStreamCall failing) + : delegate_(delegate), failing_(failing) {} + + using FileSystem::Open; + + Result> Open(const std::string& path) const override { + return delegate_->Open(path); + } + Result> Create(const std::string& path, + bool overwrite) const override { + PAIMON_ASSIGN_OR_RAISE(std::unique_ptr out, + delegate_->Create(path, overwrite)); + return std::unique_ptr(new FailingOutputStream(std::move(out), failing_)); + } + Status Mkdirs(const std::string& path) const override { + return delegate_->Mkdirs(path); + } + Status Rename(const std::string& src, const std::string& dst) const override { + return delegate_->Rename(src, dst); + } + Status Delete(const std::string& path, bool recursive = true) const override { + return delegate_->Delete(path, recursive); + } + Result GetFileStatus(const std::string& path) const override { + return delegate_->GetFileStatus(path); + } + Status ListDir(const std::string& directory, + std::vector* status_list) const override { + return delegate_->ListDir(directory, status_list); + } + Status ListFileStatus(const std::string& path, + std::vector* status_list) const override { + return delegate_->ListFileStatus(path, status_list); + } + Result Exists(const std::string& path) const override { + return delegate_->Exists(path); + } + + private: + class FailingOutputStream : public OutputStream { + public: + FailingOutputStream(std::unique_ptr delegate, FailingStreamCall failing) + : delegate_(std::move(delegate)), failing_(failing) {} + + Result Write(const char* buffer, int64_t size) override { + return delegate_->Write(buffer, size); + } + Status Flush() override { + if (failing_ == FailingStreamCall::kFlush) { + return Status::IOError("injected flush failure"); + } + return delegate_->Flush(); + } + Result GetPos() const override { + if (failing_ == FailingStreamCall::kGetPos) { + return Status::IOError("injected get position failure"); + } + return delegate_->GetPos(); + } + Result GetUri() const override { + return delegate_->GetUri(); + } + Status Close() override { + if (failing_ == FailingStreamCall::kClose) { + // Closed all the same, so the file it wrote can still be removed. + [[maybe_unused]] Status closed = delegate_->Close(); + return Status::IOError("injected close failure"); + } + return delegate_->Close(); + } + + private: + std::unique_ptr delegate_; + FailingStreamCall failing_; + }; + + std::shared_ptr delegate_; + FailingStreamCall failing_; +}; + +/// Creates a format table's schema on disk and loads the table. +Result> CreateTable( + const std::shared_ptr& file_system, const std::string& path, + const std::vector& partition_keys, + const std::map& extra_options = {}) { + std::map options = {{Options::TYPE, "format-table"}, + {Options::FILE_FORMAT, "parquet"}}; + for (const auto& [key, value] : extra_options) { + options[key] = value; + } + SchemaManager schema_manager(file_system, path); + PAIMON_ASSIGN_OR_RAISE( + std::unique_ptr table_schema, + schema_manager.CreateTable(MakeSchema(), partition_keys, /*primary_keys=*/{}, options)); + return FormatTable::Create(file_system, path, Identifier("db", "tbl")); +} + +/// Builds one batch of the table's columns, inserts unless other row kinds are given. +Result> MakeBatch( + const std::vector& ids, const std::vector& names, const std::string& dt, + const std::map& partition, + const std::vector& row_kinds = {}) { + arrow::Int32Builder id_builder; + arrow::StringBuilder name_builder; + arrow::StringBuilder dt_builder; + for (size_t i = 0; i < ids.size(); i++) { + PAIMON_RETURN_NOT_OK_FROM_ARROW(id_builder.Append(ids[i])); + PAIMON_RETURN_NOT_OK_FROM_ARROW(name_builder.Append(names[i])); + PAIMON_RETURN_NOT_OK_FROM_ARROW(dt_builder.Append(dt)); + } + std::shared_ptr id_array; + std::shared_ptr name_array; + std::shared_ptr dt_array; + PAIMON_RETURN_NOT_OK_FROM_ARROW(id_builder.Finish(&id_array)); + PAIMON_RETURN_NOT_OK_FROM_ARROW(name_builder.Finish(&name_array)); + PAIMON_RETURN_NOT_OK_FROM_ARROW(dt_builder.Finish(&dt_array)); + PAIMON_ASSIGN_OR_RAISE_FROM_ARROW( + std::shared_ptr struct_array, + arrow::StructArray::Make({id_array, name_array, dt_array}, MakeSchema()->fields())); + + auto c_array = std::make_unique(); + PAIMON_RETURN_NOT_OK_FROM_ARROW(arrow::ExportArray(*struct_array, c_array.get())); + RecordBatchBuilder builder(c_array.get()); + builder.SetPartition(partition); + if (!row_kinds.empty()) { + builder.SetRowKinds(row_kinds); + } + return builder.Finish(); +} + +/// Builds a one-row batch whose partition column is null, declared as `default_partition_name`. +Result> MakeBatchWithNullPartition( + const std::string& default_partition_name) { + arrow::Int32Builder id_builder; + arrow::StringBuilder name_builder; + arrow::StringBuilder dt_builder; + PAIMON_RETURN_NOT_OK_FROM_ARROW(id_builder.Append(1)); + PAIMON_RETURN_NOT_OK_FROM_ARROW(name_builder.Append("alice")); + PAIMON_RETURN_NOT_OK_FROM_ARROW(dt_builder.AppendNull()); + std::shared_ptr id_array; + std::shared_ptr name_array; + std::shared_ptr dt_array; + PAIMON_RETURN_NOT_OK_FROM_ARROW(id_builder.Finish(&id_array)); + PAIMON_RETURN_NOT_OK_FROM_ARROW(name_builder.Finish(&name_array)); + PAIMON_RETURN_NOT_OK_FROM_ARROW(dt_builder.Finish(&dt_array)); + PAIMON_ASSIGN_OR_RAISE_FROM_ARROW( + std::shared_ptr struct_array, + arrow::StructArray::Make({id_array, name_array, dt_array}, MakeSchema()->fields())); + + auto c_array = std::make_unique(); + PAIMON_RETURN_NOT_OK_FROM_ARROW(arrow::ExportArray(*struct_array, c_array.get())); + RecordBatchBuilder builder(c_array.get()); + builder.SetPartition({{"dt", default_partition_name}}); + return builder.Finish(); +} + +/// Builds one batch of `count` rows, large enough that a small split target has to cut it up. +Result> MakeManyRowBatch(int32_t count) { + std::vector ids; + std::vector names; + ids.reserve(count); + names.reserve(count); + for (int32_t i = 0; i < count; i++) { + ids.push_back(i); + names.push_back("name-" + std::to_string(i)); + } + return MakeBatch(ids, names, "20240101", {}); +} + +/// Builds one batch of `count` wide, all-different rows starting at `start_id`. A file is +/// measured by the bytes its writer has finished with, and small repeated values sit in a +/// dictionary it has not written yet. +Result> MakeWideRowBatch(int32_t count, int32_t start_id) { + constexpr size_t kNameLength = 1024; + std::vector ids; + std::vector names; + ids.reserve(count); + names.reserve(count); + for (int32_t i = 0; i < count; i++) { + const int32_t id = start_id + i; + ids.push_back(id); + names.push_back(std::to_string(id) + + std::string(kNameLength, static_cast('a' + (id % 26)))); + } + return MakeBatch(ids, names, "20240101", {}); +} + +/// The path a write stages `file_path` under: a `_temporary` directory beside where the file will +/// be published, holding a hidden name of its own. Java Paimon's `RenamingTwoPhaseOutputStream` +/// uses the same layout. +std::string StagedPath(const std::string& file_path) { + return PathUtil::JoinPath(PathUtil::GetParentDirPath(file_path), + "_temporary/.tmp.d9b7f0a2-0c11-4a35-9f6e-2f2f0f9e6c41"); +} + +/// Writes one batch and commits it, so the files become part of the table. +Status WriteAndCommit(const std::shared_ptr& table, + std::unique_ptr&& batch, bool overwrite = false, + const std::map& static_partition = {}) { + PAIMON_ASSIGN_OR_RAISE(std::unique_ptr write, + FormatTableWrite::Create(table, /*pool=*/nullptr)); + PAIMON_RETURN_NOT_OK(write->Write(std::move(batch))); + PAIMON_ASSIGN_OR_RAISE(std::vector messages, write->PrepareCommit()); + PAIMON_ASSIGN_OR_RAISE(std::unique_ptr commit, + FormatTableCommit::Create(table, overwrite, static_partition)); + return commit->Commit(messages); +} + +/// Imports a batch the reader handed out. `ASSERT_OK_AND_ASSIGN` only understands paimon's +/// `Result`, so arrow's has to be converted before a test body can use it. +Result> ImportBatch(const BatchReader::ReadBatch& batch) { + PAIMON_ASSIGN_OR_RAISE_FROM_ARROW( + std::shared_ptr record_batch, + arrow::ImportRecordBatch(batch.first.get(), batch.second.get())); + return record_batch; +} + +/// Reads every row of a plan, returning the rows as `id|name|dt` strings. +Result> ReadAll(const std::shared_ptr& table, + const std::vector>& splits, + const std::shared_ptr& predicate = nullptr, + bool enable_predicate_filter = false) { + PAIMON_ASSIGN_OR_RAISE( + std::unique_ptr read, + FormatTableRead::Create(table, /*projection=*/std::nullopt, /*pool=*/nullptr, predicate, + enable_predicate_filter)); + PAIMON_ASSIGN_OR_RAISE(std::unique_ptr reader, read->CreateReader(splits)); + std::vector rows; + while (true) { + PAIMON_ASSIGN_OR_RAISE(BatchReader::ReadBatch batch, reader->NextBatch()); + if (BatchReader::IsEofBatch(batch)) { + break; + } + PAIMON_ASSIGN_OR_RAISE_FROM_ARROW( + std::shared_ptr record_batch, + arrow::ImportRecordBatch(batch.first.get(), batch.second.get())); + // The leading field is `_VALUE_KIND`, which every `BatchReader` puts first; a format + // table has no row kinds of its own, so every row of it is an insert. + if (record_batch->schema()->field(0)->name() != SpecialFields::ValueKind().Name()) { + return Status::Invalid("a format table read must still carry the _VALUE_KIND field"); + } + auto row_kinds = checked_pointer_cast(record_batch->column(0)); + for (int64_t i = 0; i < record_batch->num_rows(); i++) { + if (row_kinds->Value(i) != RowKind::Insert()->ToByteValue()) { + return Status::Invalid("a format table read must return inserts only"); + } + } + auto ids = checked_pointer_cast(record_batch->column(1)); + auto names = checked_pointer_cast(record_batch->column(2)); + auto dts = checked_pointer_cast(record_batch->column(3)); + for (int64_t i = 0; i < record_batch->num_rows(); i++) { + rows.push_back(std::to_string(ids->Value(i)) + "|" + names->GetString(i) + "|" + + dts->GetString(i)); + } + } + reader->Close(); + return rows; +} + +} // namespace + +TEST(FormatTableTest, TestParseFormat) { + ASSERT_OK_AND_ASSIGN(FormatTable::Format parquet, FormatTable::ParseFormat("PARQUET")); + ASSERT_EQ(parquet, FormatTable::Format::PARQUET); + ASSERT_OK_AND_ASSIGN(FormatTable::Format orc, FormatTable::ParseFormat("orc")); + ASSERT_EQ(orc, FormatTable::Format::ORC); + ASSERT_EQ(FormatTable::FormatToString(FormatTable::Format::ORC), "orc"); + + // Format table formats with no reader here yet answer `NotImplemented`, which is a different + // answer from a name that is no format at all. + for (const char* format : {"csv", "text", "json", "mosaic"}) { + Result unimplemented = FormatTable::ParseFormat(format); + ASSERT_FALSE(unimplemented.ok()) << format; + ASSERT_TRUE(unimplemented.status().IsNotImplemented()) << format; + } + + Result unknown = FormatTable::ParseFormat("nonesuch"); + ASSERT_FALSE(unknown.ok()); + ASSERT_TRUE(unknown.status().IsInvalid()); +} + +TEST(FormatTableTest, TestCreateReadsOptions) { + std::unique_ptr dir = UniqueTestDirectory::Create(); + ASSERT_TRUE(dir); + ASSERT_OK_AND_ASSIGN(std::shared_ptr table, + CreateTable(dir->GetFileSystem(), dir->Str(), {"dt"})); + ASSERT_EQ(table->Location(), dir->Str()); + ASSERT_EQ(table->GetFormat(), FormatTable::Format::PARQUET); + ASSERT_EQ(table->PartitionKeys(), std::vector({"dt"})); + ASSERT_EQ(table->FileCompression(), "snappy"); + ASSERT_EQ(table->PartitionDefaultName(), "__DEFAULT_PARTITION__"); + ASSERT_EQ(table->FullName(), "db.tbl"); +} + +TEST(FormatTableTest, TestFileCompressionComesFromCoreOptions) { + // The resolution order itself is `CoreOptionsTest.TestFormatTableFileCompression`'s business; + // what matters here is that the table asks for it rather than resolving compression again. + std::unique_ptr dir = UniqueTestDirectory::Create(); + ASSERT_TRUE(dir); + ASSERT_OK_AND_ASSIGN( + std::shared_ptr table, + CreateTable(dir->GetFileSystem(), dir->Str(), {}, + {{Options::FORMAT_TABLE_FILE_COMPRESSION, "lz4"}, {"compression", "zstd"}})); + ASSERT_EQ(table->FileCompression(), "lz4"); + + std::unique_ptr default_dir = UniqueTestDirectory::Create(); + ASSERT_TRUE(default_dir); + ASSERT_OK_AND_ASSIGN(std::shared_ptr default_table, + CreateTable(default_dir->GetFileSystem(), default_dir->Str(), {})); + ASSERT_EQ(default_table->FileCompression(), "snappy"); +} + +TEST(FormatTableTest, TestFileFormatDefaultsToParquet) { + std::unique_ptr dir = UniqueTestDirectory::Create(); + ASSERT_TRUE(dir); + SchemaManager schema_manager(dir->GetFileSystem(), dir->Str()); + ASSERT_OK_AND_ASSIGN( + [[maybe_unused]] std::unique_ptr table_schema, + schema_manager.CreateTable(MakeSchema(), /*partition_keys=*/{}, + /*primary_keys=*/{}, {{Options::TYPE, "format-table"}})); + ASSERT_OK_AND_ASSIGN( + std::shared_ptr table, + FormatTable::Create(dir->GetFileSystem(), dir->Str(), Identifier("db", "tbl"))); + ASSERT_EQ(table->GetFormat(), FormatTable::Format::PARQUET); + ASSERT_EQ(table->FileCompression(), "snappy"); +} + +TEST(FormatTableTest, TestUnknownTableTypeIsRejected) { + std::unique_ptr dir = UniqueTestDirectory::Create(); + ASSERT_TRUE(dir); + SchemaManager schema_manager(dir->GetFileSystem(), dir->Str()); + // Refused at creation: read as a managed table it would look for snapshots it never had. + Result> unknown_type = schema_manager.CreateTable( + MakeSchema(), /*partition_keys=*/{}, /*primary_keys=*/{}, {{Options::TYPE, "nonesuch"}}); + ASSERT_FALSE(unknown_type.ok()); + ASSERT_TRUE(unknown_type.status().IsInvalid()); +} + +TEST(FormatTableTest, TestATableTypeThisLibraryCannotOpenIsRejectedAtCreation) { + std::unique_ptr dir = UniqueTestDirectory::Create(); + ASSERT_TRUE(dir); + SchemaManager schema_manager(dir->GetFileSystem(), dir->Str()); + // A table type paimon names but this library cannot open: refused up front, with its own + // status code and the type quoted. + Result> object_table = + schema_manager.CreateTable(MakeSchema(), /*partition_keys=*/{}, /*primary_keys=*/{}, + {{Options::TYPE, "object-table"}}); + ASSERT_FALSE(object_table.ok()); + ASSERT_TRUE(object_table.status().IsNotImplemented()) << object_table.status().ToString(); + ASSERT_NE(std::string::npos, object_table.status().ToString().find("object-table")); +} + +TEST(FormatTableTest, TestAPartitionColumnOfAnUnsupportedTypeIsRefusedUpFront) { + // A partition value makes the round trip through its column type on the way to a directory + // name and back. `BINARY` cannot, so a table partitioned by one is refused where the table is + // decided rather than at the first read or write of a table that already looked created. + std::shared_ptr binary_schema = + arrow::schema({arrow::field("id", arrow::int32()), arrow::field("name", arrow::utf8()), + arrow::field("bin", arrow::binary())}); + std::unique_ptr dir = UniqueTestDirectory::Create(); + ASSERT_TRUE(dir); + SchemaManager schema_manager(dir->GetFileSystem(), dir->Str()); + Result> created = schema_manager.CreateTable( + binary_schema, /*partition_keys=*/{"bin"}, /*primary_keys=*/{}, + {{Options::TYPE, "format-table"}, {Options::FILE_FORMAT, "parquet"}}); + ASSERT_FALSE(created.ok()); + ASSERT_NE(std::string::npos, created.status().ToString().find("cannot be partitioned")) + << created.status().ToString(); + + // And opening one another engine wrote fails the same way, not at its first read or write. + ASSERT_OK_AND_ASSIGN( + std::unique_ptr unchecked_schema, + TableSchema::Create(/*schema_id=*/0, binary_schema, /*partition_keys=*/{"bin"}, + /*primary_keys=*/{}, + {{Options::TYPE, "format-table"}, {Options::FILE_FORMAT, "parquet"}})); + std::shared_ptr data_schema = + checked_pointer_cast(std::shared_ptr(std::move(unchecked_schema))); + Result> opened = + FormatTable::Create(dir->GetFileSystem(), dir->Str(), Identifier("db", "tbl"), data_schema, + /*location_carries_paimon_metadata=*/true); + ASSERT_FALSE(opened.ok()); + ASSERT_NE(std::string::npos, opened.status().ToString().find("cannot be partitioned")) + << opened.status().ToString(); +} + +TEST(FormatTableTest, TestCatalogManagedPartitionsAreRejected) { + // The option moves partition visibility into the catalog: a directory nobody registered stops + // being part of the table. A scan here reads the directories instead, so honouring the option + // by ignoring it would return partitions the catalog never registered. + std::unique_ptr dir = UniqueTestDirectory::Create(); + ASSERT_TRUE(dir); + Result> table = CreateTable( + dir->GetFileSystem(), dir->Str(), {"dt"}, {{Options::METASTORE_PARTITIONED_TABLE, "true"}}); + ASSERT_FALSE(table.ok()); + ASSERT_TRUE(table.status().IsNotImplemented()); + + // Explicitly turning it off is the behaviour that is implemented, so it is accepted. + std::unique_ptr off_dir = UniqueTestDirectory::Create(); + ASSERT_TRUE(off_dir); + ASSERT_OK_AND_ASSIGN(std::shared_ptr off_table, + CreateTable(off_dir->GetFileSystem(), off_dir->Str(), {"dt"}, + {{Options::METASTORE_PARTITIONED_TABLE, "false"}})); + + // And the same refusal when it arrives at the call rather than in the schema. Validating the + // schema's own options alone would let this one through and then drop it, leaving a caller + // who asked for catalog-managed partitions with a scan that read the directories anyway. + Result> dynamic_table = + FormatTable::Create(off_dir->GetFileSystem(), off_dir->Str(), Identifier("db", "tbl"), + off_table->LatestSchema(), + /*location_carries_paimon_metadata=*/true, + {{Options::METASTORE_PARTITIONED_TABLE, "true"}}); + ASSERT_FALSE(dynamic_table.ok()); + ASSERT_TRUE(dynamic_table.status().IsNotImplemented()) << dynamic_table.status().ToString(); +} + +TEST(FormatTableTest, TestTheGenericEntryPointsValidateOptionsGivenAtTheCall) { + // An option a format table refuses has to be refused wherever it comes from. Every generic + // entry point merges what the call gave it over what the schema stored, so each runs the same + // checks over the merged result; otherwise a caller could set through a context what the + // schema would have rejected and have it silently dropped. + std::unique_ptr dir = UniqueTestDirectory::Create(); + ASSERT_TRUE(dir); + ASSERT_OK_AND_ASSIGN([[maybe_unused]] std::shared_ptr table, + CreateTable(dir->GetFileSystem(), dir->Str(), {"dt"})); + const std::map refused = { + {Options::METASTORE_PARTITIONED_TABLE, "true"}}; + + ScanContextBuilder scan_builder(dir->Str()); + scan_builder.SetOptions(refused); + ASSERT_OK_AND_ASSIGN(std::unique_ptr scan_context, scan_builder.Finish()); + ASSERT_NOK_WITH_MSG(TableScan::Create(std::move(scan_context)), "metastore.partitioned-table"); + + ReadContextBuilder read_builder(dir->Str()); + read_builder.SetOptions(refused); + ASSERT_OK_AND_ASSIGN(std::unique_ptr read_context, read_builder.Finish()); + ASSERT_NOK_WITH_MSG(TableRead::Create(std::move(read_context)), "metastore.partitioned-table"); + + WriteContextBuilder write_builder(dir->Str(), "test-user"); + write_builder.SetOptions(refused); + ASSERT_OK_AND_ASSIGN(std::unique_ptr write_context, write_builder.Finish()); + ASSERT_NOK_WITH_MSG(FileStoreWrite::Create(std::move(write_context)), + "metastore.partitioned-table"); + + CommitContextBuilder commit_builder(dir->Str(), "test-user"); + commit_builder.SetOptions(refused); + ASSERT_OK_AND_ASSIGN(std::unique_ptr commit_context, commit_builder.Finish()); + ASSERT_NOK_WITH_MSG(FileStoreCommit::Create(std::move(commit_context)), + "metastore.partitioned-table"); +} + +TEST(FormatTableTest, TestValueOnlyPartitionCannotBeNamedAfterThisTablesMetadata) { + // Under the value-only layout a value becomes a directory name unchanged, so `dt` of + // `schema` would be written over this table's own schema, and an overwrite would delete it. + std::unique_ptr dir = UniqueTestDirectory::Create(); + ASSERT_TRUE(dir); + ASSERT_OK_AND_ASSIGN(std::shared_ptr table, + CreateTable(dir->GetFileSystem(), dir->Str(), {"dt"}, + {{Options::FORMAT_TABLE_PARTITION_PATH_ONLY_VALUE, "true"}})); + ASSERT_TRUE(table->LocationCarriesPaimonMetadata()); + + for (const std::string& reserved : {std::string("schema"), std::string("branch")}) { + ASSERT_OK_AND_ASSIGN(std::unique_ptr write, + FormatTableWrite::Create(table, /*pool=*/nullptr)); + ASSERT_OK_AND_ASSIGN(std::unique_ptr batch, + MakeBatch({1}, {"alice"}, reserved, {{"dt", reserved}})); + SCOPED_TRACE(reserved); + ASSERT_NOK_WITH_MSG(write->Write(std::move(batch)), "own metadata rather than data"); + ASSERT_OK(write->Abort()); + } + + // An overwrite names its partition itself, so it is refused on its own account, before it + // deletes anything. + ASSERT_OK_AND_ASSIGN(std::unique_ptr commit, + FormatTableCommit::Create(table, /*overwrite=*/true, {{"dt", "schema"}})); + ASSERT_NOK_WITH_MSG(commit->Commit({}), "own metadata rather than data"); + // The schema is still where the table keeps it. + SchemaManager schema_manager(dir->GetFileSystem(), dir->Str()); + ASSERT_OK_AND_ASSIGN(std::optional> latest, + schema_manager.Latest()); + ASSERT_TRUE(latest.has_value()); + + // Any other value is ordinary data and still works. + ASSERT_OK_AND_ASSIGN(std::unique_ptr ordinary, + MakeBatch({2}, {"bob"}, "schematic", {{"dt", "schematic"}})); + ASSERT_OK(WriteAndCommit(table, std::move(ordinary))); + ASSERT_OK_AND_ASSIGN( + std::unique_ptr scan, + FormatTableScan::Create(table, /*partition_filter=*/{}, /*limit=*/std::nullopt)); + ASSERT_OK_AND_ASSIGN(std::shared_ptr plan, scan->CreatePlan()); + ASSERT_OK_AND_ASSIGN(std::vector rows, ReadAll(table, plan->Splits())); + ASSERT_EQ(rows, (std::vector{"2|bob|schematic"})); +} + +TEST(FormatTableTest, TestValueOnlyPartitionCannotBeNamedByAHiddenValue) { + // Under the value-only layout the partition value is the whole directory name, so a value + // starting with `_` or `.` names a directory every scan skips: the rows would be written and + // never read back. The write is refused instead of losing them quietly. + std::unique_ptr dir = UniqueTestDirectory::Create(); + ASSERT_TRUE(dir); + ASSERT_OK_AND_ASSIGN(std::shared_ptr table, + CreateTable(dir->GetFileSystem(), dir->Str(), {"dt"}, + {{Options::FORMAT_TABLE_PARTITION_PATH_ONLY_VALUE, "true"}})); + for (const std::string& hidden : {std::string("_2025"), std::string(".2025")}) { + SCOPED_TRACE(hidden); + ASSERT_OK_AND_ASSIGN(std::unique_ptr write, + FormatTableWrite::Create(table, /*pool=*/nullptr)); + ASSERT_OK_AND_ASSIGN(std::unique_ptr batch, + MakeBatch({1}, {"alice"}, hidden, {{"dt", hidden}})); + ASSERT_NOK_WITH_MSG(write->Write(std::move(batch)), "a scan of this table would skip"); + ASSERT_OK(write->Abort()); + } + + // The one hidden name that is table content: the directory standing for a null partition + // value, which this layout writes and reads like any other. What the null itself reads back + // as is `TestNullPartitionValue`'s business. + ASSERT_OK_AND_ASSIGN(std::unique_ptr null_partition, + MakeBatchWithNullPartition(table->PartitionDefaultName())); + ASSERT_OK(WriteAndCommit(table, std::move(null_partition))); + ASSERT_OK_AND_ASSIGN(bool default_dir_exists, dir->GetFileSystem()->Exists(PathUtil::JoinPath( + dir->Str(), table->PartitionDefaultName()))); + ASSERT_TRUE(default_dir_exists); + ASSERT_OK_AND_ASSIGN( + std::unique_ptr scan, + FormatTableScan::Create(table, /*partition_filter=*/{}, /*limit=*/std::nullopt)); + ASSERT_OK_AND_ASSIGN(std::shared_ptr plan, scan->CreatePlan()); + ASSERT_OK_AND_ASSIGN(std::vector rows, ReadAll(table, plan->Splits())); + ASSERT_EQ(rows.size(), 1u); + + // The same value under the `key=value` layout is ordinary data: the key in front of it makes + // the directory `dt=_2025`, which is not hidden at all. + std::unique_ptr key_value_dir = UniqueTestDirectory::Create(); + ASSERT_TRUE(key_value_dir); + ASSERT_OK_AND_ASSIGN(std::shared_ptr key_value_table, + CreateTable(key_value_dir->GetFileSystem(), key_value_dir->Str(), {"dt"})); + ASSERT_OK_AND_ASSIGN(std::unique_ptr key_value_batch, + MakeBatch({1}, {"alice"}, "_2025", {{"dt", "_2025"}})); + ASSERT_OK(WriteAndCommit(key_value_table, std::move(key_value_batch))); + ASSERT_OK_AND_ASSIGN( + std::unique_ptr key_value_scan, + FormatTableScan::Create(key_value_table, /*partition_filter=*/{}, /*limit=*/std::nullopt)); + ASSERT_OK_AND_ASSIGN(std::shared_ptr key_value_plan, key_value_scan->CreatePlan()); + ASSERT_OK_AND_ASSIGN(std::vector key_value_rows, + ReadAll(key_value_table, key_value_plan->Splits())); + ASSERT_EQ(key_value_rows, (std::vector{"1|alice|_2025"})); +} + +TEST(FormatTableTest, TestCommitMessageCannotPublishIntoThisTablesMetadata) { + // The same rule applies to a message that no writer here produced, since a commit checks the + // message it is given rather than where it came from. + std::unique_ptr dir = UniqueTestDirectory::Create(); + ASSERT_TRUE(dir); + ASSERT_OK_AND_ASSIGN(std::shared_ptr table, + CreateTable(dir->GetFileSystem(), dir->Str(), {})); + ASSERT_OK_AND_ASSIGN( + std::unique_ptr commit, + FormatTableCommit::Create(table, /*overwrite=*/false, /*static_partition=*/{})); + + FormatCommitMessage into_metadata(StagedPath(dir->Str() + "/schema/data-a-0.parquet"), + dir->Str() + "/schema/data-a-0.parquet", + std::map{}, + /*record_count=*/1, /*file_size=*/1); + ASSERT_NOK_WITH_MSG(commit->Commit({into_metadata}), "own metadata rather than data"); +} + +TEST(FormatTableTest, TestLocationBoundsAreCheckedFromTheRightComponent) { + std::unique_ptr dir = UniqueTestDirectory::Create(); + ASSERT_TRUE(dir); + SchemaManager schema_manager(dir->GetFileSystem(), dir->Str()); + ASSERT_OK_AND_ASSIGN(std::unique_ptr table_schema, + schema_manager.CreateTable( + MakeSchema(), /*partition_keys=*/{}, /*primary_keys=*/{}, + {{Options::TYPE, "format-table"}, {Options::FILE_FORMAT, "parquet"}})); + std::shared_ptr data_schema = + checked_pointer_cast(std::shared_ptr(std::move(table_schema))); + + // Every path is checked against the location, and an empty one is a prefix of nothing: it + // would either pass every path, including those outside the table, or fail them all. + ASSERT_NOK_WITH_MSG( + FormatTable::Create(dir->GetFileSystem(), "", Identifier("db", "tbl"), data_schema, + /*location_carries_paimon_metadata=*/false), + "requires a location"); + + // The file system root is its own separator, so what is below it starts one character in. + // Reading it two characters in would take `/data.parquet` for `ata.parquet`, and + // `/.hidden.parquet` for a name that is not hidden at all. + ASSERT_OK_AND_ASSIGN( + std::shared_ptr root_table, + FormatTable::Create(dir->GetFileSystem(), "/", Identifier("db", "tbl"), data_schema, + /*location_carries_paimon_metadata=*/false)); + ASSERT_EQ(root_table->Location(), "/"); + ASSERT_OK_AND_ASSIGN( + std::unique_ptr read, + FormatTableRead::Create(root_table, /*projection=*/std::nullopt, /*pool=*/nullptr, + /*predicate=*/nullptr, + /*enable_predicate_filter=*/false)); + + auto visible = std::make_shared( + std::vector{{"/data.parquet", 0}}, + std::map{}); + ASSERT_OK(read->CreateReader(std::static_pointer_cast(visible))); + + auto hidden = std::make_shared( + std::vector{{"/.data.parquet", 0}}, + std::map{}); + ASSERT_NOK_WITH_MSG(read->CreateReader(std::static_pointer_cast(hidden)), "would skip"); + + // A location written with a trailing separator names the same directory as one without, and + // bounds the same paths. + ASSERT_OK_AND_ASSIGN(std::shared_ptr trailing, + FormatTable::Create(dir->GetFileSystem(), dir->Str() + "/", + Identifier("db", "tbl"), data_schema, + /*location_carries_paimon_metadata=*/false)); + ASSERT_OK_AND_ASSIGN( + std::unique_ptr trailing_read, + FormatTableRead::Create(trailing, /*projection=*/std::nullopt, /*pool=*/nullptr, + /*predicate=*/nullptr, + /*enable_predicate_filter=*/false)); + auto inside = std::make_shared( + std::vector{{dir->Str() + "/data.parquet", 0}}, + std::map{}); + ASSERT_OK(trailing_read->CreateReader(std::static_pointer_cast(inside))); + auto outside = std::make_shared( + std::vector{{dir->Str() + "-sibling/data.parquet", 0}}, + std::map{}); + ASSERT_NOK_WITH_MSG(trailing_read->CreateReader(std::static_pointer_cast(outside)), + "not under the table location"); +} + +TEST(FormatTableTest, TestATrailingSeparatorStillNamesTheTableLocation) { + std::unique_ptr dir = UniqueTestDirectory::Create(); + ASSERT_TRUE(dir); + ASSERT_OK_AND_ASSIGN([[maybe_unused]] std::shared_ptr created, + CreateTable(dir->GetFileSystem(), dir->Str(), {})); + SchemaManager schema_manager(dir->GetFileSystem(), dir->Str()); + ASSERT_OK_AND_ASSIGN(std::optional> latest, + schema_manager.Latest()); + ASSERT_TRUE(latest.has_value()); + std::shared_ptr data_schema = checked_pointer_cast(latest.value()); + + // The same table with a trailing separator. Whether `schema` below the location is metadata + // is decided by comparing the two, so both spellings must compare equal; otherwise an + // overwrite at the root would list the schema as data and delete it. + ASSERT_OK_AND_ASSIGN(std::shared_ptr table, + FormatTable::Create(dir->GetFileSystem(), dir->Str() + "/", + Identifier("db", "tbl"), data_schema, + /*location_carries_paimon_metadata=*/true)); + + ASSERT_OK_AND_ASSIGN(std::unique_ptr batch, + MakeBatch({1}, {"alice"}, "20240101", {})); + ASSERT_OK(WriteAndCommit(table, std::move(batch), /*overwrite=*/true)); + + // The schema is still where the table keeps it, and the rows are readable. + ASSERT_OK_AND_ASSIGN(std::optional> after, + schema_manager.Latest()); + ASSERT_TRUE(after.has_value()); + ASSERT_OK_AND_ASSIGN( + std::unique_ptr scan, + FormatTableScan::Create(table, /*partition_filter=*/{}, /*limit=*/std::nullopt)); + ASSERT_OK_AND_ASSIGN(std::shared_ptr plan, scan->CreatePlan()); + ASSERT_OK_AND_ASSIGN(std::vector rows, ReadAll(table, plan->Splits())); + ASSERT_EQ(rows, (std::vector{"1|alice|20240101"})); +} + +TEST(FormatTableTest, TestExternalLocationHasNoReservedDirectories) { + // A format table served by a catalog that keeps schemas elsewhere has nothing but data below + // its location. A value-only partition whose value happens to be `schema` is such data, and + // skipping it would drop rows without a word. + std::unique_ptr schema_dir = UniqueTestDirectory::Create(); + ASSERT_TRUE(schema_dir); + std::map options = { + {Options::TYPE, "format-table"}, + {Options::FILE_FORMAT, "parquet"}, + {Options::FORMAT_TABLE_PARTITION_PATH_ONLY_VALUE, "true"}}; + SchemaManager schema_manager(schema_dir->GetFileSystem(), schema_dir->Str()); + ASSERT_OK_AND_ASSIGN(std::unique_ptr table_schema, + schema_manager.CreateTable(MakeSchema(), /*partition_keys=*/{"dt"}, + /*primary_keys=*/{}, options)); + + // The data lives somewhere else entirely, the way an external table's does. + std::unique_ptr data_dir = UniqueTestDirectory::Create(); + ASSERT_TRUE(data_dir); + std::shared_ptr data_schema = + checked_pointer_cast(std::shared_ptr(std::move(table_schema))); + ASSERT_OK_AND_ASSIGN(std::shared_ptr external, + FormatTable::Create(data_dir->GetFileSystem(), data_dir->Str(), + Identifier("db", "tbl"), data_schema, + /*location_carries_paimon_metadata=*/false)); + ASSERT_FALSE(external->LocationCarriesPaimonMetadata()); + + ASSERT_OK_AND_ASSIGN(std::unique_ptr batch, + MakeBatch({1}, {"alice"}, "schema", {{"dt", "schema"}})); + ASSERT_OK(WriteAndCommit(external, std::move(batch))); + + ASSERT_OK_AND_ASSIGN( + std::unique_ptr scan, + FormatTableScan::Create(external, /*partition_filter=*/{}, /*limit=*/std::nullopt)); + ASSERT_OK_AND_ASSIGN(std::shared_ptr plan, scan->CreatePlan()); + ASSERT_OK_AND_ASSIGN(std::vector rows, ReadAll(external, plan->Splits())); + ASSERT_EQ(rows, (std::vector{"1|alice|schema"})); +} + +TEST(FormatTableTest, TestFileSuffixIncludesCompressionWhenAsked) { + std::unique_ptr dir = UniqueTestDirectory::Create(); + ASSERT_TRUE(dir); + // A format that records its own compression keeps a plain name unless the option asks for it, + // and then the compression goes in front of the format: `data--0.snappy.parquet`. + ASSERT_OK_AND_ASSIGN(std::shared_ptr table, + CreateTable(dir->GetFileSystem(), dir->Str(), {}, + {{Options::FILE_FORMAT, "parquet"}, + {Options::FILE_SUFFIX_INCLUDE_COMPRESSION, "true"}})); + ASSERT_OK_AND_ASSIGN(std::unique_ptr write, + FormatTableWrite::Create(table, /*pool=*/nullptr)); + ASSERT_OK_AND_ASSIGN(std::unique_ptr batch, + MakeBatch({1}, {"alice"}, "20240101", {})); + ASSERT_OK(write->Write(std::move(batch))); + ASSERT_OK_AND_ASSIGN(std::vector messages, write->PrepareCommit()); + ASSERT_EQ(messages.size(), 1u); + ASSERT_TRUE(StringUtils::EndsWith(messages[0].file_path, ".snappy.parquet")) + << messages[0].file_path; + ASSERT_OK(write->Abort()); + + // A compression hadoop does not name goes into the file name as the option spelled it. + ASSERT_OK_AND_ASSIGN(std::shared_ptr uncompressed, + CreateTable(dir->GetFileSystem(), dir->Str() + "/other", {}, + {{Options::FILE_FORMAT, "parquet"}, + {Options::FILE_COMPRESSION, "uncompressed"}, + {Options::FILE_SUFFIX_INCLUDE_COMPRESSION, "true"}})); + ASSERT_OK_AND_ASSIGN(std::unique_ptr other_write, + FormatTableWrite::Create(uncompressed, /*pool=*/nullptr)); + ASSERT_OK_AND_ASSIGN(batch, MakeBatch({1}, {"alice"}, "20240101", {})); + ASSERT_OK(other_write->Write(std::move(batch))); + ASSERT_OK_AND_ASSIGN(messages, other_write->PrepareCommit()); + ASSERT_EQ(messages.size(), 1u); + ASSERT_TRUE(StringUtils::EndsWith(messages[0].file_path, ".uncompressed.parquet")) + << messages[0].file_path; + ASSERT_OK(other_write->Abort()); +} + +TEST(FormatTableTest, TestValueOnlyLayoutNeedsPartitionKeys) { + // The layout names a directory by its partition value alone, so a table with no partition + // keys asks for a layout that has nothing to lay out. + std::unique_ptr dir = UniqueTestDirectory::Create(); + ASSERT_TRUE(dir); + ASSERT_NOK_WITH_MSG(CreateTable(dir->GetFileSystem(), dir->Str(), {}, + {{Options::FORMAT_TABLE_PARTITION_PATH_ONLY_VALUE, "true"}}), + "on a table with no partition keys"); +} + +TEST(FormatTableTest, TestCreateRejectsManagedTable) { + std::unique_ptr dir = UniqueTestDirectory::Create(); + ASSERT_TRUE(dir); + SchemaManager schema_manager(dir->GetFileSystem(), dir->Str()); + ASSERT_OK_AND_ASSIGN( + [[maybe_unused]] std::unique_ptr table_schema, + schema_manager.CreateTable(MakeSchema(), /*partition_keys=*/{}, /*primary_keys=*/{}, + {{Options::FILE_FORMAT, "parquet"}})); + Result> table = + FormatTable::Create(dir->GetFileSystem(), dir->Str(), Identifier("db", "tbl")); + ASSERT_FALSE(table.ok()); + ASSERT_TRUE(table.status().IsInvalid()); +} + +TEST(FormatTableTest, TestValueOnlyPartitionLayout) { + std::unique_ptr dir = UniqueTestDirectory::Create(); + ASSERT_TRUE(dir); + ASSERT_OK_AND_ASSIGN(std::shared_ptr table, + CreateTable(dir->GetFileSystem(), dir->Str(), {"dt"}, + {{Options::FORMAT_TABLE_PARTITION_PATH_ONLY_VALUE, "true"}})); + ASSERT_TRUE(table->PartitionOnlyValueInPath()); + ASSERT_OK_AND_ASSIGN(std::unique_ptr batch, + MakeBatch({1, 2}, {"alice", "bob"}, "20240101", {{"dt", "20240101"}})); + ASSERT_OK(WriteAndCommit(table, std::move(batch))); + + // The directory is the bare value, with no field name in it. + ASSERT_OK_AND_ASSIGN(bool value_only_dir_exists, + dir->GetFileSystem()->Exists(PathUtil::JoinPath(dir->Str(), "20240101"))); + ASSERT_TRUE(value_only_dir_exists); + ASSERT_OK_AND_ASSIGN(bool key_value_dir_exists, dir->GetFileSystem()->Exists(PathUtil::JoinPath( + dir->Str(), "dt=20240101"))); + ASSERT_FALSE(key_value_dir_exists); + + // And the scan reads back the layout the write produced. + ASSERT_OK_AND_ASSIGN( + std::unique_ptr scan, + FormatTableScan::Create(table, /*partition_filter=*/{}, /*limit=*/std::nullopt)); + ASSERT_OK_AND_ASSIGN(PartitionList partitions, scan->ListPartitions()); + ASSERT_EQ(partitions.size(), 1); + ASSERT_EQ(partitions[0].at("dt"), "20240101"); + ASSERT_OK_AND_ASSIGN(std::shared_ptr plan, scan->CreatePlan()); + ASSERT_OK_AND_ASSIGN(std::vector rows, ReadAll(table, plan->Splits())); + ASSERT_EQ(rows.size(), 2); + ASSERT_EQ(rows[0], "1|alice|20240101"); +} + +TEST(FormatTableTest, TestScanFindsDataFilesInSubdirectories) { + std::unique_ptr dir = UniqueTestDirectory::Create(); + ASSERT_TRUE(dir); + ASSERT_OK_AND_ASSIGN(std::shared_ptr table, + CreateTable(dir->GetFileSystem(), dir->Str(), {})); + + // A real data file, written by this table and then moved: `data-file.path-directory` puts + // data files a level down, and an engine writing the directory may do the same. The partition + // columns are not in the file, so nesting changes nothing about what one holds. + ASSERT_OK_AND_ASSIGN(std::unique_ptr batch, + MakeBatch({1, 2}, {"alice", "bob"}, "20240101", {})); + ASSERT_OK(WriteAndCommit(table, std::move(batch))); + ASSERT_OK_AND_ASSIGN( + std::unique_ptr written, + FormatTableScan::Create(table, /*partition_filter=*/{}, /*limit=*/std::nullopt)); + ASSERT_OK_AND_ASSIGN(std::shared_ptr written_plan, written->CreatePlan()); + ASSERT_EQ(written_plan->Splits().size(), 1u); + auto written_split = std::dynamic_pointer_cast(written_plan->Splits()[0]); + ASSERT_NE(written_split, nullptr); + ASSERT_EQ(written_split->files.size(), 1u); + const std::string written_path = written_split->files[0].file_path; + + std::string nested = PathUtil::JoinPath(dir->Str(), "bucket-0"); + ASSERT_OK(dir->GetFileSystem()->Mkdirs(nested)); + const std::string nested_path = PathUtil::JoinPath(nested, PathUtil::GetName(written_path)); + ASSERT_OK(dir->GetFileSystem()->Rename(written_path, nested_path)); + + // A staging tree at the same level stays invisible, whatever its files are called, even when + // the file in it is a perfectly readable copy of the one above. + std::string staging = PathUtil::JoinPath(dir->Str(), "_temporary"); + ASSERT_OK(dir->GetFileSystem()->Mkdirs(staging)); + std::string content; + ASSERT_OK(dir->GetFileSystem()->ReadFile(nested_path, &content)); + ASSERT_OK(dir->GetFileSystem()->WriteFile( + PathUtil::JoinPath(staging, PathUtil::GetName(written_path)), content, + /*overwrite=*/true)); + + ASSERT_OK_AND_ASSIGN( + std::unique_ptr scan, + FormatTableScan::Create(table, /*partition_filter=*/{}, /*limit=*/std::nullopt)); + ASSERT_OK_AND_ASSIGN(std::shared_ptr plan, scan->CreatePlan()); + ASSERT_OK_AND_ASSIGN(std::vector rows, ReadAll(table, plan->Splits())); + ASSERT_EQ(rows.size(), 2u); + ASSERT_EQ(rows[0], "1|alice|20240101"); + ASSERT_EQ(rows[1], "2|bob|20240101"); +} + +TEST(FormatTableTest, TestTargetFileRowNumRollsFiles) { + std::unique_ptr dir = UniqueTestDirectory::Create(); + ASSERT_TRUE(dir); + ASSERT_OK_AND_ASSIGN( + std::shared_ptr table, + CreateTable(dir->GetFileSystem(), dir->Str(), {}, {{Options::TARGET_FILE_ROW_NUM, "1"}})); + ASSERT_OK_AND_ASSIGN(std::unique_ptr write, + FormatTableWrite::Create(table, /*pool=*/nullptr)); + for (int32_t i = 0; i < 3; i++) { + ASSERT_OK_AND_ASSIGN(std::unique_ptr batch, + MakeBatch({i}, {"name"}, "20240101", {})); + ASSERT_OK(write->Write(std::move(batch))); + } + ASSERT_OK_AND_ASSIGN(std::vector messages, write->PrepareCommit()); + // Rolling is checked between batches, so each one-row batch closes its own file. + ASSERT_EQ(messages.size(), 3); + ASSERT_OK_AND_ASSIGN( + std::unique_ptr commit, + FormatTableCommit::Create(table, /*overwrite=*/false, /*static_partition=*/{})); + ASSERT_OK(commit->Commit(messages)); +} + +TEST(FormatTableTest, TestTargetFileSizeRollsFiles) { + constexpr int32_t kBatches = 2; + constexpr int32_t kRowsPerBatch = 2000; + std::unique_ptr dir = UniqueTestDirectory::Create(); + ASSERT_TRUE(dir); + // Rolling is checked between batches, so each batch is weighed against the target and a file + // closes once it is past it. Without this a write would put a whole partition in one file + // however large it grew. + ASSERT_OK_AND_ASSIGN( + std::shared_ptr table, + CreateTable(dir->GetFileSystem(), dir->Str(), {}, {{Options::TARGET_FILE_SIZE, "1 kb"}})); + ASSERT_OK_AND_ASSIGN(std::unique_ptr write, + FormatTableWrite::Create(table, /*pool=*/nullptr)); + // Megabytes of values, each of them different: a file is weighed by what its writer has + // finished with, and a writer holds on to what it can still encode more cheaply later. + for (int32_t batch = 0; batch < kBatches; batch++) { + ASSERT_OK_AND_ASSIGN(std::unique_ptr rows, + MakeWideRowBatch(kRowsPerBatch, batch * kRowsPerBatch)); + ASSERT_OK(write->Write(std::move(rows))); + } + ASSERT_OK_AND_ASSIGN(std::vector messages, write->PrepareCommit()); + ASSERT_GT(messages.size(), 1u); + ASSERT_OK_AND_ASSIGN( + std::unique_ptr commit, + FormatTableCommit::Create(table, /*overwrite=*/false, /*static_partition=*/{})); + ASSERT_OK(commit->Commit(messages)); + + ASSERT_OK_AND_ASSIGN( + std::unique_ptr scan, + FormatTableScan::Create(table, /*partition_filter=*/{}, /*limit=*/std::nullopt)); + ASSERT_OK_AND_ASSIGN(std::shared_ptr plan, scan->CreatePlan()); + ASSERT_OK_AND_ASSIGN(std::vector read_rows, ReadAll(table, plan->Splits())); + ASSERT_EQ(read_rows.size(), static_cast(kBatches) * static_cast(kRowsPerBatch)); +} + +TEST(FormatTableTest, TestANonPositiveLimitPlansNothing) { + std::unique_ptr dir = UniqueTestDirectory::Create(); + ASSERT_TRUE(dir); + ASSERT_OK_AND_ASSIGN(std::shared_ptr table, + CreateTable(dir->GetFileSystem(), dir->Str(), {})); + ASSERT_OK_AND_ASSIGN(std::unique_ptr batch, + MakeBatch({1, 2}, {"alice", "bob"}, "20240101", {})); + ASSERT_OK(WriteAndCommit(table, std::move(batch))); + + // A limit of zero asks for no rows, so there is nothing to read and no split to hand out. A + // positive limit cannot drop anything, since a format table records no row counts. + ASSERT_OK_AND_ASSIGN(std::unique_ptr none, + FormatTableScan::Create(table, /*partition_filter=*/{}, /*limit=*/0)); + ASSERT_OK_AND_ASSIGN(std::shared_ptr empty_plan, none->CreatePlan()); + ASSERT_TRUE(empty_plan->Splits().empty()); + + ASSERT_OK_AND_ASSIGN(std::unique_ptr some, + FormatTableScan::Create(table, /*partition_filter=*/{}, /*limit=*/1)); + ASSERT_OK_AND_ASSIGN(std::shared_ptr plan, some->CreatePlan()); + ASSERT_FALSE(plan->Splits().empty()); +} + +TEST(FormatTableTest, TestSplitsAreBoundedByTheSplitTargetSize) { + std::unique_ptr dir = UniqueTestDirectory::Create(); + ASSERT_TRUE(dir); + // One file per split: every file costs at least the open-file cost, which fills a split on its + // own. Without packing the whole partition would be a single split however many files it has. + ASSERT_OK_AND_ASSIGN(std::shared_ptr table, + CreateTable(dir->GetFileSystem(), dir->Str(), {}, + {{Options::TARGET_FILE_ROW_NUM, "1"}, + {Options::SOURCE_SPLIT_TARGET_SIZE, "1 kb"}, + {Options::SOURCE_SPLIT_OPEN_FILE_COST, "1 kb"}})); + ASSERT_OK_AND_ASSIGN(std::unique_ptr write, + FormatTableWrite::Create(table, /*pool=*/nullptr)); + for (int32_t i = 0; i < 3; i++) { + ASSERT_OK_AND_ASSIGN(std::unique_ptr batch, + MakeBatch({i}, {"name"}, "20240101", {})); + ASSERT_OK(write->Write(std::move(batch))); + } + ASSERT_OK_AND_ASSIGN(std::vector messages, write->PrepareCommit()); + ASSERT_OK_AND_ASSIGN( + std::unique_ptr commit, + FormatTableCommit::Create(table, /*overwrite=*/false, /*static_partition=*/{})); + ASSERT_OK(commit->Commit(messages)); + + ASSERT_OK_AND_ASSIGN( + std::unique_ptr scan, + FormatTableScan::Create(table, /*partition_filter=*/{}, /*limit=*/std::nullopt)); + ASSERT_OK_AND_ASSIGN(std::shared_ptr plan, scan->CreatePlan()); + ASSERT_EQ(plan->Splits().size(), 3); + // Every row is still read exactly once, whichever split it landed in. + ASSERT_OK_AND_ASSIGN(std::vector rows, ReadAll(table, plan->Splits())); + ASSERT_EQ(rows.size(), 3); +} + +TEST(FormatTableTest, TestAFileLargerThanTheSplitTargetIsStillOneSplit) { + std::unique_ptr dir = UniqueTestDirectory::Create(); + ASSERT_TRUE(dir); + // One file, far larger than a split. + ASSERT_OK_AND_ASSIGN(std::shared_ptr table, + CreateTable(dir->GetFileSystem(), dir->Str(), {}, + {{Options::SOURCE_SPLIT_TARGET_SIZE, "1 kb"}, + {Options::SOURCE_SPLIT_OPEN_FILE_COST, "1 kb"}})); + ASSERT_OK_AND_ASSIGN(std::unique_ptr batch, MakeManyRowBatch(500)); + ASSERT_OK(WriteAndCommit(table, std::move(batch))); + + ASSERT_OK_AND_ASSIGN( + std::unique_ptr scan, + FormatTableScan::Create(table, /*partition_filter=*/{}, /*limit=*/std::nullopt)); + ASSERT_OK_AND_ASSIGN(std::shared_ptr plan, scan->CreatePlan()); + // A split holds whole files, because parquet and orc record their own row group and stripe + // boundaries, so a file past the target size is one split rather than several. + ASSERT_EQ(plan->Splits().size(), 1u); + + // And every row of it is read exactly once. + ASSERT_OK_AND_ASSIGN(std::vector rows, ReadAll(table, plan->Splits())); + ASSERT_EQ(rows.size(), 500u); + std::vector expected; + expected.reserve(500); + for (int32_t i = 0; i < 500; i++) { + expected.push_back(std::to_string(i) + "|name-" + std::to_string(i) + "|20240101"); + } + std::sort(rows.begin(), rows.end()); + std::sort(expected.begin(), expected.end()); + ASSERT_EQ(rows, expected); +} + +TEST(FormatTableTest, TestBlankPartitionValueLandsInTheDefaultPartition) { + std::unique_ptr dir = UniqueTestDirectory::Create(); + ASSERT_TRUE(dir); + ASSERT_OK_AND_ASSIGN(std::shared_ptr table, + CreateTable(dir->GetFileSystem(), dir->Str(), {"dt"})); + // A null, an empty string and a whitespace-only string alike stand for the default partition + // name. Treating only null that way would put the other two in directories of their own, and + // an empty one has no legal directory name at all. + ASSERT_OK_AND_ASSIGN( + std::unique_ptr blank, + MakeBatch({1, 2}, {"alice", "bob"}, " ", {{"dt", table->PartitionDefaultName()}})); + ASSERT_OK(WriteAndCommit(table, std::move(blank))); + + ASSERT_OK_AND_ASSIGN( + std::unique_ptr scan, + FormatTableScan::Create(table, /*partition_filter=*/{}, /*limit=*/std::nullopt)); + ASSERT_OK_AND_ASSIGN(std::shared_ptr plan, scan->CreatePlan()); + ASSERT_OK_AND_ASSIGN(PartitionList partitions, scan->ListPartitions()); + ASSERT_EQ(partitions.size(), 1u); + ASSERT_EQ(partitions[0].at("dt"), table->PartitionDefaultName()); + // The value reads back as null, as it does for a null partition. + ASSERT_OK_AND_ASSIGN(std::vector rows, ReadAll(table, plan->Splits())); + ASSERT_EQ(rows.size(), 2u); +} + +TEST(FormatTableTest, TestAFileThatCannotBeClosedEndsTheWrite) { + // A store can refuse `Flush()`, `GetPos()` or `Close()` on the stream a file is written + // through, and each lands at a different point: opening the file, adding to it, or closing it. + // Wherever it lands, the write answers with the failure and hands out nothing to publish, + // rather than reaching through a writer that is no longer there. + for (const FailingStreamCall failing : + {FailingStreamCall::kGetPos, FailingStreamCall::kFlush, FailingStreamCall::kClose}) { + SCOPED_TRACE(static_cast(failing)); + // Rolling on every row, so a refusal in the closing path lands inside `Write()`. + std::unique_ptr dir = UniqueTestDirectory::Create(); + ASSERT_TRUE(dir); + ASSERT_OK_AND_ASSIGN([[maybe_unused]] std::shared_ptr created, + CreateTable(dir->GetFileSystem(), dir->Str(), {}, + {{Options::TARGET_FILE_ROW_NUM, "1"}})); + auto failing_fs = std::make_shared(dir->GetFileSystem(), failing); + ASSERT_OK_AND_ASSIGN(std::shared_ptr table, + FormatTable::Create(failing_fs, dir->Str(), Identifier("db", "tbl"))); + + ASSERT_OK_AND_ASSIGN(std::unique_ptr write, + FormatTableWrite::Create(table, /*pool=*/nullptr)); + ASSERT_OK_AND_ASSIGN(std::unique_ptr first, + MakeBatch({1}, {"alice"}, "20240101", {})); + ASSERT_NOK(write->Write(std::move(first))); + + ASSERT_OK_AND_ASSIGN(std::unique_ptr second, + MakeBatch({2}, {"bob"}, "20240101", {})); + ASSERT_NOK(write->Write(std::move(second))); + // A refusal while closing ends the write, since the rows of a file that cannot be closed + // can never be published. One while the file was being opened staged nothing at all, so + // there is nothing to refuse and nothing to hand out either. + Result> prepared = write->PrepareCommit(); + if (prepared.ok()) { + ASSERT_TRUE(prepared.value().empty()); + } + ASSERT_OK(write->Abort()); + + // Whatever it gave up on is gone, so a later scan does not see a partial file. + ASSERT_OK_AND_ASSIGN( + std::shared_ptr plain_table, + FormatTable::Create(dir->GetFileSystem(), dir->Str(), Identifier("db", "tbl"))); + ASSERT_OK_AND_ASSIGN( + std::unique_ptr scan, + FormatTableScan::Create(plain_table, /*partition_filter=*/{}, /*limit=*/std::nullopt)); + ASSERT_OK_AND_ASSIGN(std::shared_ptr plan, scan->CreatePlan()); + ASSERT_TRUE(plan->Splits().empty()); + } + + // `Close()` is reached only once the writer has been finished and dropped, so a refusal there + // is the case worth pinning down: the write is over, and both entry points say so rather than + // publishing what is left or reaching through the writer that is gone. + std::unique_ptr dir = UniqueTestDirectory::Create(); + ASSERT_TRUE(dir); + ASSERT_OK_AND_ASSIGN([[maybe_unused]] std::shared_ptr created, + CreateTable(dir->GetFileSystem(), dir->Str(), {})); + auto failing_fs = + std::make_shared(dir->GetFileSystem(), FailingStreamCall::kClose); + ASSERT_OK_AND_ASSIGN(std::shared_ptr table, + FormatTable::Create(failing_fs, dir->Str(), Identifier("db", "tbl"))); + ASSERT_OK_AND_ASSIGN(std::unique_ptr write, + FormatTableWrite::Create(table, /*pool=*/nullptr)); + ASSERT_OK_AND_ASSIGN(std::unique_ptr batch, + MakeBatch({1}, {"alice"}, "20240101", {})); + ASSERT_OK(write->Write(std::move(batch))); + ASSERT_NOK(write->PrepareCommit()); + ASSERT_NOK(write->PrepareCommit()); + ASSERT_OK_AND_ASSIGN(std::unique_ptr after, + MakeBatch({2}, {"bob"}, "20240101", {})); + ASSERT_NOK(write->Write(std::move(after))); + ASSERT_OK(write->Abort()); +} + +TEST(FormatTableTest, TestAFailedOpenClosesTheFileBeforeDeletingIt) { + std::unique_ptr dir = UniqueTestDirectory::Create(); + ASSERT_TRUE(dir); + // A compression no codec answers to, so opening the file succeeds and building the writer + // over it fails: the one path where a temp file exists and nothing owns it yet. + ASSERT_OK_AND_ASSIGN([[maybe_unused]] std::shared_ptr created, + CreateTable(dir->GetFileSystem(), dir->Str(), {}, + {{Options::FILE_COMPRESSION, "nonesuch"}})); + auto recording = std::make_shared(dir->GetFileSystem()); + ASSERT_OK_AND_ASSIGN(std::shared_ptr table, + FormatTable::Create(recording, dir->Str(), Identifier("db", "tbl"))); + + ASSERT_OK_AND_ASSIGN(std::unique_ptr write, + FormatTableWrite::Create(table, /*pool=*/nullptr)); + ASSERT_OK_AND_ASSIGN(std::unique_ptr batch, + MakeBatch({1}, {"alice"}, "20240101", {})); + Status status = write->Write(std::move(batch)); + ASSERT_FALSE(status.ok()) << "a writer was built over a compression nothing implements"; + + // Created, closed, and only then deleted. Deleting first would leave the stream to flush + // afterwards, and on a store that writes from its destructor the file would come back under + // a hidden name no scan reads and no abort knows about. + const std::vector& calls = recording->Calls(); + ASSERT_EQ(calls.size(), 3u); + ASSERT_TRUE(StringUtils::StartsWith(calls[0], "create ")) << calls[0]; + ASSERT_TRUE(StringUtils::StartsWith(calls[1], "close ")) << calls[1]; + ASSERT_TRUE(StringUtils::StartsWith(calls[2], "delete ")) << calls[2]; + // All three name the one temp file, which is hidden so that no scan can reach it. + const std::string temp_path = calls[0].substr(std::string("create ").size()); + ASSERT_EQ(calls[1], "close " + temp_path); + ASSERT_EQ(calls[2], "delete " + temp_path); + ASSERT_TRUE(StringUtils::StartsWith(PathUtil::GetName(temp_path), ".")) << temp_path; + + // Nothing is left behind for a later scan to pick up. + ASSERT_OK_AND_ASSIGN(bool exists, dir->GetFileSystem()->Exists(temp_path)); + ASSERT_FALSE(exists); +} + +TEST(FormatTableTest, TestCreateRejectsSchemasThatCouldNeverBeUsed) { + std::unique_ptr dir = UniqueTestDirectory::Create(); + ASSERT_TRUE(dir); + // Every one of these can be persisted and loaded, and would otherwise only fail when a reader + // or writer is built, long after the table looked created. + ASSERT_NOK_WITH_MSG(CreateTable(dir->GetFileSystem(), dir->Str() + "/a", {}, + {{Options::FILE_FORMAT, "nonesuch"}}), + "unsupported file format"); + ASSERT_NOK_WITH_MSG(CreateTable(dir->GetFileSystem(), dir->Str() + "/b", {}, + {{Options::TARGET_FILE_ROW_NUM, "0"}}), + "should be at least 1"); + // A format table format with no reader here is refused by name, rather than failing later + // with a missing-format-factory error. + ASSERT_NOK_WITH_MSG( + CreateTable(dir->GetFileSystem(), dir->Str() + "/c", {}, {{Options::FILE_FORMAT, "csv"}}), + "not supported by paimon-cpp yet"); + // Every column a partition column leaves the data files with nothing in them. + ASSERT_NOK_WITH_MSG( + CreateTable(dir->GetFileSystem(), dir->Str() + "/d", {"id", "name", "dt"}, {}), + "every one of its columns"); +} + +TEST(FormatTableTest, TestPrimaryKeysAreRejectedWhenTheTableIsCreated) { + std::unique_ptr dir = UniqueTestDirectory::Create(); + ASSERT_TRUE(dir); + SchemaManager schema_manager(dir->GetFileSystem(), dir->Str()); + // A format table with primary keys could never be opened, so it is never written either. + Result> table_schema = schema_manager.CreateTable( + MakeSchema(), /*partition_keys=*/{}, /*primary_keys=*/{"id"}, + {{Options::TYPE, "format-table"}, {Options::FILE_FORMAT, "parquet"}}); + ASSERT_FALSE(table_schema.ok()); + ASSERT_TRUE(table_schema.status().IsInvalid()); +} + +TEST(FormatTableTest, TestWriteReadUnpartitioned) { + std::unique_ptr dir = UniqueTestDirectory::Create(); + ASSERT_TRUE(dir); + ASSERT_OK_AND_ASSIGN(std::shared_ptr table, + CreateTable(dir->GetFileSystem(), dir->Str(), {})); + ASSERT_OK_AND_ASSIGN(std::unique_ptr batch, + MakeBatch({1, 2}, {"alice", "bob"}, "20240101", {})); + ASSERT_OK(WriteAndCommit(table, std::move(batch))); + + ASSERT_OK_AND_ASSIGN( + std::unique_ptr scan, + FormatTableScan::Create(table, /*partition_filter=*/{}, /*limit=*/std::nullopt)); + ASSERT_OK_AND_ASSIGN(std::shared_ptr plan, scan->CreatePlan()); + ASSERT_EQ(plan->Splits().size(), 1); + + ASSERT_OK_AND_ASSIGN(std::vector rows, ReadAll(table, plan->Splits())); + ASSERT_EQ(rows.size(), 2u); + ASSERT_EQ(rows[0], "1|alice|20240101"); + ASSERT_EQ(rows[1], "2|bob|20240101"); +} + +TEST(FormatTableTest, TestBatchesOfOnePartitionShareOneFile) { + // The directory a partition writes into is derived once and kept, so a second batch for the + // same partition finds the file the first one opened instead of starting another. A partition + // that has not been seen before still gets a file of its own. + std::unique_ptr dir = UniqueTestDirectory::Create(); + ASSERT_TRUE(dir); + ASSERT_OK_AND_ASSIGN(std::shared_ptr table, + CreateTable(dir->GetFileSystem(), dir->Str(), {"dt"})); + ASSERT_OK_AND_ASSIGN(std::unique_ptr write, + FormatTableWrite::Create(table, /*pool=*/nullptr)); + for (const auto& [id, name, dt] : std::vector>{ + {1, "alice", "20240101"}, {2, "bob", "20240101"}, {3, "carol", "20240102"}}) { + ASSERT_OK_AND_ASSIGN(std::unique_ptr batch, + MakeBatch({id}, {name}, dt, {{"dt", dt}})); + ASSERT_OK(write->Write(std::move(batch))); + } + + ASSERT_OK_AND_ASSIGN(std::vector messages, write->PrepareCommit()); + // One file per partition, not one per batch. + ASSERT_EQ(messages.size(), 2u); + std::sort(messages.begin(), messages.end(), + [](const FormatCommitMessage& left, const FormatCommitMessage& right) { + return left.file_path < right.file_path; + }); + ASSERT_EQ(messages[0].partition, (std::map{{"dt", "20240101"}})); + ASSERT_EQ(messages[0].record_count, 2); + ASSERT_EQ(messages[1].partition, (std::map{{"dt", "20240102"}})); + ASSERT_EQ(messages[1].record_count, 1); + + ASSERT_OK_AND_ASSIGN( + std::unique_ptr commit, + FormatTableCommit::Create(table, /*overwrite=*/false, /*static_partition=*/{})); + ASSERT_OK(commit->Commit(messages)); + ASSERT_OK_AND_ASSIGN( + std::unique_ptr scan, + FormatTableScan::Create(table, /*partition_filter=*/{}, /*limit=*/std::nullopt)); + ASSERT_OK_AND_ASSIGN(std::shared_ptr plan, scan->CreatePlan()); + ASSERT_OK_AND_ASSIGN(std::vector rows, ReadAll(table, plan->Splits())); + std::sort(rows.begin(), rows.end()); + ASSERT_EQ(rows, + (std::vector{"1|alice|20240101", "2|bob|20240101", "3|carol|20240102"})); +} + +TEST(FormatTableTest, TestWriteReadPartitioned) { + std::unique_ptr dir = UniqueTestDirectory::Create(); + ASSERT_TRUE(dir); + ASSERT_OK_AND_ASSIGN(std::shared_ptr table, + CreateTable(dir->GetFileSystem(), dir->Str(), {"dt"})); + ASSERT_OK_AND_ASSIGN(std::unique_ptr first, + MakeBatch({1}, {"alice"}, "20240101", {{"dt", "20240101"}})); + ASSERT_OK(WriteAndCommit(table, std::move(first))); + ASSERT_OK_AND_ASSIGN(std::unique_ptr second, + MakeBatch({2}, {"bob"}, "20240102", {{"dt", "20240102"}})); + ASSERT_OK(WriteAndCommit(table, std::move(second))); + + // Each partition is its own directory, so each is its own split. + ASSERT_OK_AND_ASSIGN( + std::unique_ptr scan, + FormatTableScan::Create(table, /*partition_filter=*/{}, /*limit=*/std::nullopt)); + ASSERT_OK_AND_ASSIGN(std::shared_ptr plan, scan->CreatePlan()); + ASSERT_EQ(plan->Splits().size(), 2); + + // The partition column is rebuilt from the directory name, not read from the file. + ASSERT_OK_AND_ASSIGN(std::vector rows, ReadAll(table, plan->Splits())); + ASSERT_EQ(rows.size(), 2); + ASSERT_EQ(rows[0], "1|alice|20240101"); + ASSERT_EQ(rows[1], "2|bob|20240102"); + + ASSERT_OK_AND_ASSIGN(PartitionList partitions, scan->ListPartitions()); + ASSERT_EQ(partitions.size(), 2); +} + +TEST(FormatTableTest, TestPartitionsAreListedInAStableOrder) { + std::unique_ptr dir = UniqueTestDirectory::Create(); + ASSERT_TRUE(dir); + ASSERT_OK_AND_ASSIGN(std::shared_ptr table, + CreateTable(dir->GetFileSystem(), dir->Str(), {"dt"})); + // Written out of order, because the listing order must not depend on it: the file system + // promises none, and a caller comparing two listings would see partitions move. + for (const char* dt : {"20240103", "20240101", "20240102"}) { + ASSERT_OK_AND_ASSIGN(std::unique_ptr batch, + MakeBatch({1}, {"alice"}, dt, {{"dt", dt}})); + ASSERT_OK(WriteAndCommit(table, std::move(batch))); + } + + ASSERT_OK_AND_ASSIGN( + std::unique_ptr scan, + FormatTableScan::Create(table, /*partition_filter=*/{}, /*limit=*/std::nullopt)); + ASSERT_OK_AND_ASSIGN(PartitionList partitions, scan->ListPartitions()); + ASSERT_EQ(partitions.size(), 3u); + ASSERT_EQ(partitions[0].at("dt"), "20240101"); + ASSERT_EQ(partitions[1].at("dt"), "20240102"); + ASSERT_EQ(partitions[2].at("dt"), "20240103"); +} + +TEST(FormatTableTest, TestPartitionFilter) { + std::unique_ptr dir = UniqueTestDirectory::Create(); + ASSERT_TRUE(dir); + ASSERT_OK_AND_ASSIGN(std::shared_ptr table, + CreateTable(dir->GetFileSystem(), dir->Str(), {"dt"})); + ASSERT_OK_AND_ASSIGN(std::unique_ptr first, + MakeBatch({1}, {"alice"}, "20240101", {{"dt", "20240101"}})); + ASSERT_OK(WriteAndCommit(table, std::move(first))); + ASSERT_OK_AND_ASSIGN(std::unique_ptr second, + MakeBatch({2}, {"bob"}, "20240102", {{"dt", "20240102"}})); + ASSERT_OK(WriteAndCommit(table, std::move(second))); + + ASSERT_OK_AND_ASSIGN( + std::unique_ptr scan, + FormatTableScan::Create(table, {{"dt", "20240102"}}, /*limit=*/std::nullopt)); + ASSERT_OK_AND_ASSIGN(std::shared_ptr plan, scan->CreatePlan()); + ASSERT_EQ(plan->Splits().size(), 1); + ASSERT_OK_AND_ASSIGN(std::vector rows, ReadAll(table, plan->Splits())); + ASSERT_EQ(rows.size(), 1); + ASSERT_EQ(rows[0], "2|bob|20240102"); +} + +TEST(FormatTableTest, TestScanRejectsUnknownPartitionField) { + std::unique_ptr dir = UniqueTestDirectory::Create(); + ASSERT_TRUE(dir); + ASSERT_OK_AND_ASSIGN(std::shared_ptr table, + CreateTable(dir->GetFileSystem(), dir->Str(), {"dt"})); + Result> scan = + FormatTableScan::Create(table, {{"name", "alice"}}, /*limit=*/std::nullopt); + ASSERT_FALSE(scan.ok()); + ASSERT_TRUE(scan.status().IsInvalid()); +} + +TEST(FormatTableTest, TestTheGenericEntryPointsReachAFormatTable) { + // A caller holding a table path uses the interfaces it uses for every other table. Each one + // recognises a format table from its schema and dispatches to it, the way Java Paimon serves + // both kinds through one `ReadBuilder` and one `BatchWriteBuilder`. + std::unique_ptr dir = UniqueTestDirectory::Create(); + ASSERT_TRUE(dir); + ASSERT_OK_AND_ASSIGN([[maybe_unused]] std::shared_ptr table, + CreateTable(dir->GetFileSystem(), dir->Str(), {"dt"})); + + WriteContextBuilder write_builder(dir->Str(), "test-user"); + ASSERT_OK_AND_ASSIGN(std::unique_ptr write_context, write_builder.Finish()); + ASSERT_OK_AND_ASSIGN(std::unique_ptr write, + FileStoreWrite::Create(std::move(write_context))); + ASSERT_OK_AND_ASSIGN(std::unique_ptr batch, + MakeBatch({1, 2}, {"alice", "bob"}, "20240101", {{"dt", "20240101"}})); + ASSERT_OK(write->Write(std::move(batch))); + ASSERT_OK_AND_ASSIGN(std::vector> messages, + write->PrepareCommit()); + ASSERT_EQ(messages.size(), 1u); + // Compaction is about manifests and buckets, so it says what it cannot do rather than + // reporting success for work it never did. + ASSERT_NOK_WITH_MSG(write->Compact({{"dt", "20240101"}}, /*bucket=*/0, + /*full_compaction=*/false), + "cannot be compacted"); + + // A write id prefixes a postpone-bucket writer's files; a format table has no buckets. + WriteContextBuilder write_id_builder(dir->Str(), "test-user"); + write_id_builder.WithWriteId(3); + ASSERT_OK_AND_ASSIGN(std::unique_ptr write_id_context, write_id_builder.Finish()); + ASSERT_NOK_WITH_MSG(FileStoreWrite::Create(std::move(write_id_context)), + "a write id would name nothing"); + + // The three `CommitContext` settings that describe snapshot machinery. Each is refused only + // when it is set away from its default, so an ordinary commit is unaffected. + CommitContextBuilder empty_commit_builder(dir->Str(), "test-user"); + empty_commit_builder.IgnoreEmptyCommit(false); + ASSERT_OK_AND_ASSIGN(std::unique_ptr empty_commit_context, + empty_commit_builder.Finish()); + ASSERT_NOK_WITH_MSG(FileStoreCommit::Create(std::move(empty_commit_context)), + "cannot record an empty commit"); + + CommitContextBuilder rest_builder(dir->Str(), "test-user"); + rest_builder.UseRESTCatalogCommit(true); + ASSERT_OK_AND_ASSIGN(std::unique_ptr rest_context, rest_builder.Finish()); + ASSERT_NOK_WITH_MSG(FileStoreCommit::Create(std::move(rest_context)), "rest catalog"); + + CommitContextBuilder conflict_builder(dir->Str(), "test-user"); + conflict_builder.AppendCommitCheckConflict(true); + ASSERT_OK_AND_ASSIGN(std::unique_ptr conflict_context, + conflict_builder.Finish()); + ASSERT_NOK_WITH_MSG(FileStoreCommit::Create(std::move(conflict_context)), + "no manifests to check a concurrent commit against"); + + CommitContextBuilder commit_builder(dir->Str(), "test-user"); + ASSERT_OK_AND_ASSIGN(std::unique_ptr commit_context, commit_builder.Finish()); + ASSERT_OK_AND_ASSIGN(std::unique_ptr commit, + FileStoreCommit::Create(std::move(commit_context))); + ASSERT_OK(commit->Commit(messages)); + // The same for the snapshot half of the commit interface. + ASSERT_NOK_WITH_MSG(commit->Expire(), "no snapshots to expire"); + // Closing after a prepared commit must not take back what the commit just published. + ASSERT_OK(write->Close()); + + ScanContextBuilder scan_builder(dir->Str()); + ASSERT_OK_AND_ASSIGN(std::unique_ptr scan_context, scan_builder.Finish()); + ASSERT_OK_AND_ASSIGN(std::unique_ptr scan, + TableScan::Create(std::move(scan_context))); + ASSERT_OK_AND_ASSIGN(std::shared_ptr plan, scan->CreatePlan()); + ASSERT_EQ(plan->Splits().size(), 1u); + + ReadContextBuilder read_builder(dir->Str()); + ASSERT_OK_AND_ASSIGN(std::unique_ptr read_context, read_builder.Finish()); + ASSERT_OK_AND_ASSIGN(std::unique_ptr read, + TableRead::Create(std::move(read_context))); + ASSERT_OK_AND_ASSIGN(std::unique_ptr reader, read->CreateReader(plan->Splits())); + ASSERT_OK_AND_ASSIGN(BatchReader::ReadBatch read_batch, reader->NextBatch()); + ASSERT_FALSE(BatchReader::IsEofBatch(read_batch)); + ASSERT_OK_AND_ASSIGN(std::shared_ptr record_batch, ImportBatch(read_batch)); + // `_VALUE_KIND` first, then the table's own columns, exactly as the narrower interface gives + // them. + ASSERT_EQ(record_batch->num_columns(), 4); + ASSERT_EQ(record_batch->num_rows(), 2); + ASSERT_EQ(record_batch->schema()->field(0)->name(), SpecialFields::ValueKind().Name()); + reader->Close(); +} + +TEST(FormatTableTest, TestTheGenericEntryPointsCarryTheContextThrough) { + // A context promises that options given at the call win over the ones the schema stored, and + // that a branch and a caller-held schema are honoured. Dispatching to a format table must not + // quietly drop any of that. + std::unique_ptr dir = UniqueTestDirectory::Create(); + ASSERT_TRUE(dir); + ASSERT_OK_AND_ASSIGN(std::shared_ptr table, + CreateTable(dir->GetFileSystem(), dir->Str(), {})); + + // `target-file-row-num` is stored nowhere in this table's schema, so a file would hold every + // row. Given at the call it has to roll a file per row, as it does for a managed table. + WriteContextBuilder write_builder(dir->Str(), "test-user"); + write_builder.SetOptions({{Options::TARGET_FILE_ROW_NUM, "1"}}); + ASSERT_OK_AND_ASSIGN(std::unique_ptr write_context, write_builder.Finish()); + ASSERT_OK_AND_ASSIGN(std::unique_ptr write, + FileStoreWrite::Create(std::move(write_context))); + for (int32_t i = 0; i < 3; i++) { + ASSERT_OK_AND_ASSIGN(std::unique_ptr batch, + MakeBatch({i}, {"name"}, "20240101", {})); + ASSERT_OK(write->Write(std::move(batch))); + } + ASSERT_OK_AND_ASSIGN(std::vector> messages, + write->PrepareCommit()); + ASSERT_EQ(messages.size(), 3u) << "target-file-row-num given at the call was ignored"; + + // Published through the generic commit, or nothing below would see the rows: until the commit + // renames them the files sit in the hidden `_temporary` directory a scan skips. + CommitContextBuilder generic_commit_builder(dir->Str(), "test-user"); + ASSERT_OK_AND_ASSIGN(std::unique_ptr generic_commit_context, + generic_commit_builder.Finish()); + ASSERT_OK_AND_ASSIGN(std::unique_ptr generic_commit, + FileStoreCommit::Create(std::move(generic_commit_context))); + ASSERT_OK(generic_commit->Commit(messages)); + ASSERT_OK(write->Close()); + + // A branch this table has no schema on is a managed table as far as dispatch is concerned, so + // the format branch must not answer for it. + ReadContextBuilder branch_builder(dir->Str()); + branch_builder.WithBranch("nosuchbranch"); + ASSERT_OK_AND_ASSIGN(std::unique_ptr branch_context, branch_builder.Finish()); + ASSERT_NOK(TableRead::Create(std::move(branch_context))); + + // A caller-held schema is used instead of reading one from under the path. + SchemaManager schema_manager(dir->GetFileSystem(), dir->Str()); + ASSERT_OK_AND_ASSIGN(std::optional> latest, + schema_manager.Latest()); + ASSERT_TRUE(latest.has_value()); + ASSERT_OK_AND_ASSIGN(std::string schema_json, (*latest)->GetJsonSchema()); + ScanContextBuilder seeded_builder(dir->Str()); + seeded_builder.SetTableSchema(schema_json); + ASSERT_OK_AND_ASSIGN(std::unique_ptr seeded_context, seeded_builder.Finish()); + ASSERT_OK_AND_ASSIGN(std::unique_ptr seeded_scan, + TableScan::Create(std::move(seeded_context))); + // Planning, not just dispatch: handing the schema over must not turn the `schema` directory + // the table keeps under its own path into data, which a plan would then read as a data file. + ASSERT_OK_AND_ASSIGN(std::shared_ptr seeded_plan, seeded_scan->CreatePlan()); + ASSERT_OK_AND_ASSIGN(std::vector seeded_rows, + ReadAll(table, seeded_plan->Splits())); + ASSERT_EQ(seeded_rows.size(), 3u); + + // `type` is structural, so an option given at the call must not decide what kind of table + // this is. Checked by reading the rows back rather than by counting splits: three small files + // are packed into one split, so a split count says nothing about how many there are. + ScanContextBuilder retyped_builder(dir->Str()); + retyped_builder.SetOptions({{Options::TYPE, "table"}}); + ASSERT_OK_AND_ASSIGN(std::unique_ptr retyped_context, retyped_builder.Finish()); + ASSERT_OK_AND_ASSIGN(std::unique_ptr retyped_scan, + TableScan::Create(std::move(retyped_context))); + ASSERT_OK_AND_ASSIGN(std::shared_ptr retyped_plan, retyped_scan->CreatePlan()); + ASSERT_OK_AND_ASSIGN(std::vector retyped_rows, + ReadAll(table, retyped_plan->Splits())); + ASSERT_EQ(retyped_rows.size(), 3u); +} + +namespace { + +/// Counts the lookups a read makes, which is enough to tell that the cache a context carries +/// reached the format reader rather than being dropped on the way; parquet looks its footer up +/// here. +class CountingCache : public Cache { + public: + Result> Get( + const std::shared_ptr& key, + std::function>(const std::shared_ptr&)> + supplier) override { + gets++; + return supplier(key); + } + Status Put(const std::shared_ptr&, const std::shared_ptr&) override { + return Status::OK(); + } + void Invalidate(const std::shared_ptr&) override {} + void InvalidateAll() override {} + size_t Size() const override { + return 0; + } + + int32_t gets = 0; +}; + +} // namespace + +TEST(FormatTableTest, TestTheGenericReadCarriesPrefetchAndTheCacheThrough) { + // A format table opens its files through the same component the managed table path opens its + // own with, so what a `ReadContext` asks for about opening a file applies here too. + std::unique_ptr dir = UniqueTestDirectory::Create(); + ASSERT_TRUE(dir); + ASSERT_OK_AND_ASSIGN(std::shared_ptr table, + CreateTable(dir->GetFileSystem(), dir->Str(), {"dt"})); + ASSERT_OK_AND_ASSIGN( + std::unique_ptr batch, + MakeBatch({1, 2, 3}, {"alice", "bob", "carol"}, "20240101", {{"dt", "20240101"}})); + ASSERT_OK(WriteAndCommit(table, std::move(batch))); + + ScanContextBuilder scan_builder(dir->Str()); + ASSERT_OK_AND_ASSIGN(std::unique_ptr scan_context, scan_builder.Finish()); + ASSERT_OK_AND_ASSIGN(std::unique_ptr scan, + TableScan::Create(std::move(scan_context))); + ASSERT_OK_AND_ASSIGN(std::shared_ptr plan, scan->CreatePlan()); + + auto read_rows = [&plan](std::unique_ptr context) -> Result { + PAIMON_ASSIGN_OR_RAISE(std::unique_ptr read, + TableRead::Create(std::move(context))); + PAIMON_ASSIGN_OR_RAISE(std::unique_ptr reader, + read->CreateReader(plan->Splits())); + int64_t rows = 0; + while (true) { + PAIMON_ASSIGN_OR_RAISE(BatchReader::ReadBatch read_batch, reader->NextBatch()); + if (BatchReader::IsEofBatch(read_batch)) { + break; + } + PAIMON_ASSIGN_OR_RAISE_FROM_ARROW( + std::shared_ptr record_batch, + arrow::ImportRecordBatch(read_batch.first.get(), read_batch.second.get())); + rows += record_batch->num_rows(); + } + reader->Close(); + return rows; + }; + + // Prefetch is honoured rather than refused, and reads back what was written. + ReadContextBuilder prefetch_builder(dir->Str()); + prefetch_builder.EnablePrefetch(true); + ASSERT_OK_AND_ASSIGN(std::unique_ptr prefetch_context, prefetch_builder.Finish()); + ASSERT_OK_AND_ASSIGN(int64_t prefetched_rows, read_rows(std::move(prefetch_context))); + ASSERT_EQ(prefetched_rows, 3); + + // And the cache the context carries reaches the format reader. + auto cache = std::make_shared(); + ReadContextBuilder cache_builder(dir->Str()); + cache_builder.WithCache(cache); + ASSERT_OK_AND_ASSIGN(std::unique_ptr cache_context, cache_builder.Finish()); + ASSERT_OK_AND_ASSIGN(int64_t cached_rows, read_rows(std::move(cache_context))); + ASSERT_EQ(cached_rows, 3); + ASSERT_GT(cache->gets, 0) << "the cache the read context carries never reached the file reader"; +} + +TEST(FormatTableTest, TestTheGenericEntryPointsRefuseWhatTheyCannotHonour) { + // Anything a context carries that a format table cannot do is a refusal naming the setting, + // never a read or a write that quietly did something else. + std::unique_ptr dir = UniqueTestDirectory::Create(); + ASSERT_TRUE(dir); + ASSERT_OK_AND_ASSIGN(std::shared_ptr table, + CreateTable(dir->GetFileSystem(), dir->Str(), {"dt"})); + + // A projected read schema can rename a column, prune a nested one and give it metadata of its + // own; a format table's projection is a list of top-level names. + ReadContextBuilder read_schema_builder(dir->Str()); + ASSERT_OK_AND_ASSIGN(std::unique_ptr<::ArrowSchema> projected_schema, table->GetArrowSchema()); + read_schema_builder.SetReadSchema(std::move(projected_schema)); + ASSERT_OK_AND_ASSIGN(std::unique_ptr read_schema_context, + read_schema_builder.Finish()); + ASSERT_NOK_WITH_MSG(TableRead::Create(std::move(read_schema_context)), + "does not take a projected read schema"); + + ScanContextBuilder streaming_builder(dir->Str()); + streaming_builder.WithStreamingMode(true); + ASSERT_OK_AND_ASSIGN(std::unique_ptr streaming_context, + streaming_builder.Finish()); + ASSERT_NOK_WITH_MSG(TableScan::Create(std::move(streaming_context)), + "nothing for a streaming scan to follow"); + + ScanContextBuilder predicate_builder(dir->Str()); + predicate_builder.SetPredicate(PredicateBuilder::GreaterThan( + /*field_index=*/0, /*field_name=*/"id", FieldType::INT, Literal(1))); + ASSERT_OK_AND_ASSIGN(std::unique_ptr predicate_context, + predicate_builder.Finish()); + ASSERT_NOK_WITH_MSG(TableScan::Create(std::move(predicate_context)), + "does not take a predicate"); + + CommitContextBuilder commit_builder(dir->Str(), "test-user"); + ASSERT_OK_AND_ASSIGN(std::unique_ptr commit_context, commit_builder.Finish()); + ASSERT_OK_AND_ASSIGN(std::unique_ptr commit, + FileStoreCommit::Create(std::move(commit_context))); + // A commit takes batch writes only. The rest of the snapshot half is + // `TestTheGenericCommitRefusesEveryCallAboutSnapshots`. + ASSERT_NOK_WITH_MSG(commit->Commit({}, /*commit_identifier=*/7), "commit identifier"); + ASSERT_NOK_WITH_MSG(commit->Commit({}, BATCH_WRITE_COMMIT_IDENTIFIER, /*watermark=*/1), + "watermark"); + + // A commit message describing manifest files is not published just because the interface + // takes the base type. `CommitMessage` has no public subclass a test can build, so this + // stands in for one: anything that is not a `FormatCommitMessage` is refused. + class NotAFormatMessage : public CommitMessage {}; + ASSERT_NOK_WITH_MSG(commit->Commit({std::make_shared()}), + "describes files to record in a manifest"); +} + +TEST(FormatTableTest, TestTheGenericCommitRefusesEveryCallAboutSnapshots) { + // Each of these describes snapshot or manifest state a format table does not keep, and each + // says so rather than reporting success for work it never did. A caller moving between table + // types then finds out at the call rather than from a table that did not change. + std::unique_ptr dir = UniqueTestDirectory::Create(); + ASSERT_TRUE(dir); + ASSERT_OK_AND_ASSIGN([[maybe_unused]] std::shared_ptr table, + CreateTable(dir->GetFileSystem(), dir->Str(), {"dt"})); + CommitContextBuilder commit_builder(dir->Str(), "test-user"); + ASSERT_OK_AND_ASSIGN(std::unique_ptr commit_context, commit_builder.Finish()); + ASSERT_OK_AND_ASSIGN(std::unique_ptr commit, + FileStoreCommit::Create(std::move(commit_context))); + + ASSERT_NOK_WITH_MSG(commit->CommitWithProgress({}, BATCH_WRITE_COMMIT_IDENTIFIER, + /*watermark=*/std::nullopt), + "real-time offsets"); + ASSERT_NOK_WITH_MSG(commit->FilterAndCommit({}), "which commit identifiers"); + ASSERT_NOK_WITH_MSG( + commit->FilterAndOverwrite({{"dt", "20240101"}}, {}, BATCH_WRITE_COMMIT_IDENTIFIER), + "which commit identifiers"); + ASSERT_NOK_WITH_MSG(commit->GetLastCommitTableRequest(), "rest catalog"); + ASSERT_NOK_WITH_MSG(commit->Expire(), "no snapshots to expire"); + ASSERT_NOK_WITH_MSG(commit->RollbackToAsLatest(/*target_snapshot_id=*/1), "roll back to"); + ASSERT_NOK_WITH_MSG( + commit->DropPartition({{{"dt", "20240101"}}}, BATCH_WRITE_COMMIT_IDENTIFIER), + "dropping a partition"); + ASSERT_NOK_WITH_MSG(commit->TruncateTable(BATCH_WRITE_COMMIT_IDENTIFIER), + "emptying a format table"); + + // The one call that cannot refuse, since it returns a reference: it has to be a no-op that + // hands back the same commit. A format table records no row ids, so there is no conflict. + ASSERT_EQ(&commit->RowIdCheckConflict(/*row_id_check_from_snapshot=*/std::nullopt), + commit.get()); + // Empty rather than null, so a caller merging metrics need not tell a table type that keeps + // none apart from one that does. + ASSERT_NE(commit->GetCommitMetrics(), nullptr); +} + +TEST(FormatTableTest, TestTheGenericCommitOverwritesOnePartition) { + // `FileStoreCommit::Overwrite()` names the partition to replace, which is the generic form of + // a static-partition overwrite: it clears that directory and leaves the others alone. + std::unique_ptr dir = UniqueTestDirectory::Create(); + ASSERT_TRUE(dir); + ASSERT_OK_AND_ASSIGN(std::shared_ptr table, + CreateTable(dir->GetFileSystem(), dir->Str(), {"dt"})); + ASSERT_OK_AND_ASSIGN(std::unique_ptr first, + MakeBatch({1, 2}, {"alice", "bob"}, "20240101", {{"dt", "20240101"}})); + ASSERT_OK(WriteAndCommit(table, std::move(first))); + ASSERT_OK_AND_ASSIGN(std::unique_ptr other, + MakeBatch({9}, {"zoe"}, "20240102", {{"dt", "20240102"}})); + ASSERT_OK(WriteAndCommit(table, std::move(other))); + + WriteContextBuilder write_builder(dir->Str(), "test-user"); + ASSERT_OK_AND_ASSIGN(std::unique_ptr write_context, write_builder.Finish()); + ASSERT_OK_AND_ASSIGN(std::unique_ptr write, + FileStoreWrite::Create(std::move(write_context))); + ASSERT_OK_AND_ASSIGN(std::unique_ptr replacement, + MakeBatch({3}, {"carol"}, "20240101", {{"dt", "20240101"}})); + ASSERT_OK(write->Write(std::move(replacement))); + // Empty rather than null on the write side too; what this write produced is on its messages. + ASSERT_NE(write->GetMetrics(), nullptr); + ASSERT_OK_AND_ASSIGN(std::vector> messages, + write->PrepareCommit()); + ASSERT_EQ(messages.size(), 1u); + + CommitContextBuilder commit_builder(dir->Str(), "test-user"); + ASSERT_OK_AND_ASSIGN(std::unique_ptr commit_context, commit_builder.Finish()); + ASSERT_OK_AND_ASSIGN(std::unique_ptr commit, + FileStoreCommit::Create(std::move(commit_context))); + // Batch writes only, and refused before anything is cleared: an overwrite that failed on its + // arguments must not have deleted the partition it named. + ASSERT_NOK_WITH_MSG(commit->Overwrite({{"dt", "20240101"}}, messages, /*commit_identifier=*/7), + "commit identifier"); + ASSERT_NOK_WITH_MSG(commit->Overwrite({{"dt", "20240101"}}, messages, + BATCH_WRITE_COMMIT_IDENTIFIER, /*watermark=*/1), + "watermark"); + ASSERT_OK(commit->Overwrite({{"dt", "20240101"}}, messages, BATCH_WRITE_COMMIT_IDENTIFIER)); + ASSERT_OK(write->Close()); + + ASSERT_OK_AND_ASSIGN( + std::unique_ptr scan, + FormatTableScan::Create(table, /*partition_filter=*/{}, /*limit=*/std::nullopt)); + ASSERT_OK_AND_ASSIGN(std::shared_ptr plan, scan->CreatePlan()); + ASSERT_OK_AND_ASSIGN(std::vector rows, ReadAll(table, plan->Splits())); + std::sort(rows.begin(), rows.end()); + ASSERT_EQ(rows, (std::vector{"3|carol|20240101", "9|zoe|20240102"})); +} + +TEST(FormatTableTest, TestReadsCarryTheValueKindField) { + // `BatchReader::NextBatch()` promises a leading `_VALUE_KIND` field, and engines read by field + // index, so dropping it would shift every column by one. A format table records no row kind, + // so the field is there and every row is an insert. + std::unique_ptr dir = UniqueTestDirectory::Create(); + ASSERT_TRUE(dir); + ASSERT_OK_AND_ASSIGN(std::shared_ptr table, + CreateTable(dir->GetFileSystem(), dir->Str(), {"dt"})); + ASSERT_OK_AND_ASSIGN(std::unique_ptr batch, + MakeBatch({1, 2}, {"alice", "bob"}, "20240101", {{"dt", "20240101"}})); + ASSERT_OK(WriteAndCommit(table, std::move(batch))); + ASSERT_OK_AND_ASSIGN( + std::unique_ptr scan, + FormatTableScan::Create(table, /*partition_filter=*/{}, /*limit=*/std::nullopt)); + ASSERT_OK_AND_ASSIGN(std::shared_ptr plan, scan->CreatePlan()); + + auto check = [](const std::shared_ptr& record_batch) { + // First, and ahead of the table's own columns. + ASSERT_EQ(record_batch->num_columns(), 4); + ASSERT_EQ(record_batch->schema()->field(0)->name(), SpecialFields::ValueKind().Name()); + ASSERT_EQ(record_batch->schema()->field(1)->name(), "id"); + auto row_kinds = checked_pointer_cast(record_batch->column(0)); + ASSERT_EQ(row_kinds->null_count(), 0); + for (int64_t i = 0; i < record_batch->num_rows(); i++) { + ASSERT_EQ(row_kinds->Value(i), RowKind::Insert()->ToByteValue()); + } + }; + + ASSERT_OK_AND_ASSIGN( + std::unique_ptr read, + FormatTableRead::Create(table, /*projection=*/std::nullopt, /*pool=*/nullptr, + /*predicate=*/nullptr, + /*enable_predicate_filter=*/false)); + ASSERT_OK_AND_ASSIGN(std::unique_ptr reader, read->CreateReader(plan->Splits())); + ASSERT_OK_AND_ASSIGN(BatchReader::ReadBatch batch_read, reader->NextBatch()); + ASSERT_FALSE(BatchReader::IsEofBatch(batch_read)); + ASSERT_OK_AND_ASSIGN(std::shared_ptr record_batch, ImportBatch(batch_read)); + check(record_batch); + reader->Close(); + + // The same promise holds on the bitmap path, which is the one a caller reaches for when + // deletion vectors or indexes are in play. + ASSERT_OK_AND_ASSIGN(std::unique_ptr bitmap_reader, + read->CreateReader(plan->Splits())); + ASSERT_OK_AND_ASSIGN(BatchReader::ReadBatchWithBitmap batch_with_bitmap, + bitmap_reader->NextBatchWithBitmap()); + ASSERT_FALSE(BatchReader::IsEofBatch(batch_with_bitmap)); + ASSERT_OK_AND_ASSIGN(std::shared_ptr bitmap_batch, + ImportBatch(batch_with_bitmap.first)); + check(bitmap_batch); + bitmap_reader->Close(); +} + +TEST(FormatTableTest, TestProjectionReordersAndDropsColumns) { + std::unique_ptr dir = UniqueTestDirectory::Create(); + ASSERT_TRUE(dir); + ASSERT_OK_AND_ASSIGN(std::shared_ptr table, + CreateTable(dir->GetFileSystem(), dir->Str(), {"dt"})); + ASSERT_OK_AND_ASSIGN( + std::unique_ptr batch, + MakeBatch({1, 2, 3}, {"alice", "bob", "carol"}, "20240101", {{"dt", "20240101"}})); + ASSERT_OK(WriteAndCommit(table, std::move(batch))); + + ASSERT_OK_AND_ASSIGN( + std::unique_ptr scan, + FormatTableScan::Create(table, /*partition_filter=*/{}, /*limit=*/std::nullopt)); + ASSERT_OK_AND_ASSIGN(std::shared_ptr plan, scan->CreatePlan()); + + // A projection may name the partition column and reorder the rest. + std::vector projection = {"dt", "name"}; + ASSERT_OK_AND_ASSIGN( + std::unique_ptr read, + FormatTableRead::Create(table, projection, /*pool=*/nullptr, /*predicate=*/nullptr, + /*enable_predicate_filter=*/false)); + ASSERT_OK_AND_ASSIGN(std::unique_ptr reader, read->CreateReader(plan->Splits())); + ASSERT_OK_AND_ASSIGN(BatchReader::ReadBatch batch_read, reader->NextBatch()); + ASSERT_FALSE(BatchReader::IsEofBatch(batch_read)); + ASSERT_OK_AND_ASSIGN(std::shared_ptr record_batch, ImportBatch(batch_read)); + // The projected columns, behind the `_VALUE_KIND` field every `BatchReader` puts first. + ASSERT_EQ(record_batch->num_columns(), 3); + ASSERT_EQ(record_batch->num_rows(), 3); + ASSERT_EQ(record_batch->schema()->field(0)->name(), SpecialFields::ValueKind().Name()); + ASSERT_EQ(record_batch->schema()->field(1)->name(), "dt"); + ASSERT_EQ(record_batch->schema()->field(2)->name(), "name"); + ASSERT_EQ(checked_pointer_cast(record_batch->column(1))->GetString(0), + "20240101"); + ASSERT_EQ(checked_pointer_cast(record_batch->column(2))->GetString(1), + "bob"); + + ASSERT_OK_AND_ASSIGN(BatchReader::ReadBatch eof, reader->NextBatch()); + ASSERT_TRUE(BatchReader::IsEofBatch(eof)); + reader->Close(); +} + +TEST(FormatTableTest, TestProjectionOfPartitionColumnOnly) { + std::unique_ptr dir = UniqueTestDirectory::Create(); + ASSERT_TRUE(dir); + ASSERT_OK_AND_ASSIGN(std::shared_ptr table, + CreateTable(dir->GetFileSystem(), dir->Str(), {"dt"})); + ASSERT_OK_AND_ASSIGN(std::unique_ptr batch, + MakeBatch({1, 2}, {"alice", "bob"}, "20240101", {{"dt", "20240101"}})); + ASSERT_OK(WriteAndCommit(table, std::move(batch))); + + ASSERT_OK_AND_ASSIGN( + std::unique_ptr scan, + FormatTableScan::Create(table, /*partition_filter=*/{}, /*limit=*/std::nullopt)); + ASSERT_OK_AND_ASSIGN(std::shared_ptr plan, scan->CreatePlan()); + + // The partition value is constant, so only the file can say how many rows to repeat it for. + std::vector projection = {"dt"}; + ASSERT_OK_AND_ASSIGN( + std::unique_ptr read, + FormatTableRead::Create(table, projection, /*pool=*/nullptr, /*predicate=*/nullptr, + /*enable_predicate_filter=*/false)); + ASSERT_OK_AND_ASSIGN(std::unique_ptr reader, read->CreateReader(plan->Splits())); + ASSERT_OK_AND_ASSIGN(BatchReader::ReadBatch batch_read, reader->NextBatch()); + ASSERT_FALSE(BatchReader::IsEofBatch(batch_read)); + ASSERT_OK_AND_ASSIGN(std::shared_ptr record_batch, ImportBatch(batch_read)); + ASSERT_EQ(record_batch->num_columns(), 2); + ASSERT_EQ(record_batch->num_rows(), 2); + ASSERT_EQ(record_batch->schema()->field(0)->name(), SpecialFields::ValueKind().Name()); + ASSERT_EQ(record_batch->schema()->field(1)->name(), "dt"); + auto dts = checked_pointer_cast(record_batch->column(1)); + ASSERT_EQ(dts->GetString(0), "20240101"); + ASSERT_EQ(dts->GetString(1), "20240101"); + reader->Close(); +} + +TEST(FormatTableTest, TestNullPartitionValue) { + std::unique_ptr dir = UniqueTestDirectory::Create(); + ASSERT_TRUE(dir); + ASSERT_OK_AND_ASSIGN(std::shared_ptr table, + CreateTable(dir->GetFileSystem(), dir->Str(), {"dt"})); + // A null partition value is carried as the default partition name in the batch's partition + // spec and in the directory it names, while the column itself holds a real null. + ASSERT_OK_AND_ASSIGN(std::unique_ptr batch, + MakeBatchWithNullPartition(table->PartitionDefaultName())); + ASSERT_OK(WriteAndCommit(table, std::move(batch))); + + ASSERT_OK_AND_ASSIGN( + std::unique_ptr scan, + FormatTableScan::Create(table, /*partition_filter=*/{}, /*limit=*/std::nullopt)); + ASSERT_OK_AND_ASSIGN(std::shared_ptr plan, scan->CreatePlan()); + ASSERT_EQ(plan->Splits().size(), 1); + + ASSERT_OK_AND_ASSIGN( + std::unique_ptr read, + FormatTableRead::Create(table, /*projection=*/std::nullopt, /*pool=*/nullptr, + /*predicate=*/nullptr, + /*enable_predicate_filter=*/false)); + ASSERT_OK_AND_ASSIGN(std::unique_ptr reader, read->CreateReader(plan->Splits())); + ASSERT_OK_AND_ASSIGN(BatchReader::ReadBatch batch_read, reader->NextBatch()); + ASSERT_FALSE(BatchReader::IsEofBatch(batch_read)); + ASSERT_OK_AND_ASSIGN(std::shared_ptr record_batch, ImportBatch(batch_read)); + ASSERT_EQ(record_batch->num_rows(), 1); + // The partition column reads back as null, not as the placeholder directory name. + ASSERT_TRUE(record_batch->column(3)->IsNull(0)); + reader->Close(); +} + +TEST(FormatTableTest, TestUncommittedFilesAreInvisible) { + std::unique_ptr dir = UniqueTestDirectory::Create(); + ASSERT_TRUE(dir); + ASSERT_OK_AND_ASSIGN(std::shared_ptr table, + CreateTable(dir->GetFileSystem(), dir->Str(), {})); + ASSERT_OK_AND_ASSIGN(std::unique_ptr batch, + MakeBatch({1}, {"alice"}, "20240101", {})); + ASSERT_OK_AND_ASSIGN(std::unique_ptr write, + FormatTableWrite::Create(table, /*pool=*/nullptr)); + ASSERT_OK(write->Write(std::move(batch))); + ASSERT_OK_AND_ASSIGN(std::vector messages, write->PrepareCommit()); + ASSERT_EQ(messages.size(), 1); + + // The data is on disk under a hidden name, and a scan does not see it until it is committed. + ASSERT_OK_AND_ASSIGN(bool temp_exists, + dir->GetFileSystem()->Exists(messages[0].temp_file_path)); + ASSERT_TRUE(temp_exists); + ASSERT_OK_AND_ASSIGN( + std::unique_ptr scan, + FormatTableScan::Create(table, /*partition_filter=*/{}, /*limit=*/std::nullopt)); + ASSERT_OK_AND_ASSIGN(std::shared_ptr plan, scan->CreatePlan()); + ASSERT_TRUE(plan->Splits().empty()); + + ASSERT_OK_AND_ASSIGN( + std::unique_ptr commit, + FormatTableCommit::Create(table, /*overwrite=*/false, /*static_partition=*/{})); + ASSERT_OK(commit->Commit(messages)); + ASSERT_OK_AND_ASSIGN( + std::unique_ptr after_commit, + FormatTableScan::Create(table, /*partition_filter=*/{}, /*limit=*/std::nullopt)); + ASSERT_OK_AND_ASSIGN(std::shared_ptr committed_plan, after_commit->CreatePlan()); + ASSERT_EQ(committed_plan->Splits().size(), 1); +} + +TEST(FormatTableTest, TestWriteAbortStillCleansUpAfterPrepareCommit) { + std::unique_ptr dir = UniqueTestDirectory::Create(); + ASSERT_TRUE(dir); + ASSERT_OK_AND_ASSIGN(std::shared_ptr table, + CreateTable(dir->GetFileSystem(), dir->Str(), {})); + ASSERT_OK_AND_ASSIGN(std::unique_ptr batch, + MakeBatch({1}, {"alice"}, "20240101", {})); + ASSERT_OK_AND_ASSIGN(std::unique_ptr write, + FormatTableWrite::Create(table, /*pool=*/nullptr)); + ASSERT_OK(write->Write(std::move(batch))); + ASSERT_OK_AND_ASSIGN(std::vector messages, write->PrepareCommit()); + ASSERT_EQ(messages.size(), 1u); + + // A commit that is prepared and then abandoned still has to be cleaned up through the write + // that staged it. Handing the messages out must not leave the write with nothing to remove + // while it goes on reporting success. + ASSERT_OK(write->Abort()); + ASSERT_OK_AND_ASSIGN(bool temp_exists, + dir->GetFileSystem()->Exists(messages[0].temp_file_path)); + ASSERT_FALSE(temp_exists); +} + +TEST(FormatTableTest, TestAFinishedWriteSaysWhichWayItFinished) { + std::unique_ptr dir = UniqueTestDirectory::Create(); + ASSERT_TRUE(dir); + ASSERT_OK_AND_ASSIGN(std::shared_ptr table, + CreateTable(dir->GetFileSystem(), dir->Str(), {})); + + // A prepared write and an aborted one both take no more rows, but the caller has different + // work to do about each (commit the messages it holds, or start over), so the refusal says + // which one happened. + ASSERT_OK_AND_ASSIGN(std::unique_ptr prepared, + FormatTableWrite::Create(table, /*pool=*/nullptr)); + ASSERT_OK(prepared->PrepareCommit().status()); + ASSERT_OK_AND_ASSIGN(std::unique_ptr batch, + MakeBatch({1}, {"alice"}, "20240101", {})); + ASSERT_NOK_WITH_MSG(prepared->Write(std::move(batch)), "already prepared its commit"); + ASSERT_NOK_WITH_MSG(prepared->PrepareCommit(), "already prepared its commit"); + + ASSERT_OK_AND_ASSIGN(std::unique_ptr aborted, + FormatTableWrite::Create(table, /*pool=*/nullptr)); + ASSERT_OK(aborted->Abort()); + ASSERT_OK_AND_ASSIGN(batch, MakeBatch({1}, {"alice"}, "20240101", {})); + ASSERT_NOK_WITH_MSG(aborted->Write(std::move(batch)), "has been aborted"); + ASSERT_NOK_WITH_MSG(aborted->PrepareCommit(), "has been aborted"); + // Aborting is the one call that still works, so a caller cleaning up need not track whether + // it has already done so. + ASSERT_OK(aborted->Abort()); +} + +TEST(FormatTableTest, TestCommitRefusesAMessageFromElsewhere) { + std::unique_ptr dir = UniqueTestDirectory::Create(); + ASSERT_TRUE(dir); + ASSERT_OK_AND_ASSIGN(std::shared_ptr table, + CreateTable(dir->GetFileSystem(), dir->Str(), {})); + ASSERT_OK_AND_ASSIGN( + std::unique_ptr commit, + FormatTableCommit::Create(table, /*overwrite=*/false, /*static_partition=*/{})); + + // Committing a message renames one path and an overwrite clears the directory around it, and + // nothing downstream re-checks those paths, so a message that does not describe a file of this + // table has to be refused here. + FormatCommitMessage outside(StagedPath("/somewhere/else/data-a-0.parquet"), + "/somewhere/else/data-a-0.parquet", {}, 1, 1); + ASSERT_NOK_WITH_MSG(commit->Commit({outside}), "not under the table location"); + // `Abort()` is best effort and never fails: it refuses the message, says so in the log and + // carries on, so that one bad message cannot strand the staged files of the good ones. + ASSERT_OK(commit->Abort({outside})); + + FormatCommitMessage across_directories(StagedPath(dir->Str() + "/a/data-a-0.parquet"), + dir->Str() + "/b/data-a-0.parquet", {}, 1, 1); + ASSERT_NOK_WITH_MSG(commit->Commit({across_directories}), "does not stage its file under"); + + FormatCommitMessage not_staged(dir->Str() + "/data-a-0.parquet", + dir->Str() + "/data-a-0.parquet", {}, 1, 1); + ASSERT_NOK_WITH_MSG(commit->Commit({not_staged}), "does not stage its file under"); + + // A hidden name beside the target is how an earlier paimon-cpp staged its files, before the + // `_temporary` directory Java Paimon uses. It is still hidden, so nothing about the path says + // it is wrong; only this check does. + FormatCommitMessage beside_the_target(dir->Str() + "/.data-a-0.parquet.tmp", + dir->Str() + "/data-a-0.parquet", {}, 1, 1); + ASSERT_NOK_WITH_MSG(commit->Commit({beside_the_target}), "does not stage its file under"); + + FormatCommitMessage negative(StagedPath(dir->Str() + "/data-a-0.parquet"), + dir->Str() + "/data-a-0.parquet", {}, -1, 1); + ASSERT_NOK_WITH_MSG(commit->Commit({negative}), "negative row count"); + + // A prefix test alone would let this through: it starts with the table location and still + // resolves outside it. + FormatCommitMessage escaping(StagedPath(dir->Str() + "/../victim/data-a-0.parquet"), + dir->Str() + "/../victim/data-a-0.parquet", {}, 1, 1); + ASSERT_NOK_WITH_MSG(commit->Commit({escaping}), "does not stay inside"); +} + +TEST(FormatTableTest, TestAPartitionValueMayBeEscapedMoreThanOneWay) { + std::unique_ptr dir = UniqueTestDirectory::Create(); + ASSERT_TRUE(dir); + ASSERT_OK_AND_ASSIGN(std::shared_ptr table, + CreateTable(dir->GetFileSystem(), dir->Str(), {"dt"})); + ASSERT_OK_AND_ASSIGN( + std::unique_ptr commit, + FormatTableCommit::Create(table, /*overwrite=*/false, /*static_partition=*/{})); + + // A directory another engine wrote may spell a value differently from how this writer would: + // `100%` and `100%25` name the same value. Comparing the directory string would refuse a + // perfectly readable partition, so the values are what is compared. + FormatCommitMessage raw_percent(StagedPath(dir->Str() + "/dt=100%/data-a-0.parquet"), + dir->Str() + "/dt=100%/data-a-0.parquet", {{"dt", "100%"}}, 1, + 1); + ASSERT_OK(commit->Abort({raw_percent})); + + // A file below the partition directory is still that partition's, which is what lets a + // partition hold its data files in plain subdirectories. + FormatCommitMessage nested(StagedPath(dir->Str() + "/dt=20240101/part-0/data-a-0.parquet"), + dir->Str() + "/dt=20240101/part-0/data-a-0.parquet", + {{"dt", "20240101"}}, 1, 1); + ASSERT_OK(commit->Abort({nested})); +} + +TEST(FormatTableTest, TestCommitBindsAFileToThePartitionItClaims) { + std::unique_ptr dir = UniqueTestDirectory::Create(); + ASSERT_TRUE(dir); + ASSERT_OK_AND_ASSIGN(std::shared_ptr table, + CreateTable(dir->GetFileSystem(), dir->Str(), {"dt"})); + + // The directory is derived from the partition, never trusted: a message whose path says one + // partition and whose values say another would publish rows under a partition they never had, + // and would have an overwrite clear the wrong one. + ASSERT_OK_AND_ASSIGN( + std::unique_ptr commit, + FormatTableCommit::Create(table, /*overwrite=*/false, /*static_partition=*/{})); + FormatCommitMessage mismatched(StagedPath(dir->Str() + "/dt=20240102/data-a-0.parquet"), + dir->Str() + "/dt=20240102/data-a-0.parquet", + {{"dt", "20240101"}}, 1, 1); + ASSERT_NOK_WITH_MSG(commit->Commit({mismatched}), "but claims"); + + // A metadata directory is refused before the partition is even looked at: it is not this + // table's data at all, whatever partition the message claims for it. + FormatCommitMessage into_metadata(StagedPath(dir->Str() + "/schema/data-a-0.parquet"), + dir->Str() + "/schema/data-a-0.parquet", {{"dt", "20240101"}}, + 1, 1); + ASSERT_NOK_WITH_MSG(commit->Commit({into_metadata}), "own metadata rather than data"); + + // A directory that is neither metadata nor a partition is refused as no partition of this + // table. + FormatCommitMessage not_a_partition(StagedPath(dir->Str() + "/plain/data-a-0.parquet"), + dir->Str() + "/plain/data-a-0.parquet", + {{"dt", "20240101"}}, 1, 1); + ASSERT_NOK_WITH_MSG(commit->Commit({not_a_partition}), "is not a partition of"); + + // And a message must carry every partition key, or nothing says where it belongs. + FormatCommitMessage no_partition(StagedPath(dir->Str() + "/dt=20240101/data-a-0.parquet"), + dir->Str() + "/dt=20240101/data-a-0.parquet", {}, 1, 1); + ASSERT_NOK_WITH_MSG(commit->Commit({no_partition}), "partition values"); + + // A static partition bounds what a commit may publish. + ASSERT_OK_AND_ASSIGN( + std::unique_ptr static_commit, + FormatTableCommit::Create(table, /*overwrite=*/true, {{"dt", "20240101"}})); + FormatCommitMessage other_partition(StagedPath(dir->Str() + "/dt=20240102/data-a-0.parquet"), + dir->Str() + "/dt=20240102/data-a-0.parquet", + {{"dt", "20240102"}}, 1, 1); + ASSERT_NOK_WITH_MSG(static_commit->Commit({other_partition}), "static partition"); +} + +TEST(FormatTableTest, TestCommitRefusesAnUnpublishableMessage) { + std::unique_ptr dir = UniqueTestDirectory::Create(); + ASSERT_TRUE(dir); + ASSERT_OK_AND_ASSIGN(std::shared_ptr table, + CreateTable(dir->GetFileSystem(), dir->Str(), {})); + ASSERT_OK_AND_ASSIGN( + std::unique_ptr commit, + FormatTableCommit::Create(table, /*overwrite=*/false, /*static_partition=*/{})); + + // Publishing under a hidden name would clear the old data on an overwrite and then succeed at + // producing a file no scan will ever return. + FormatCommitMessage hidden_target(StagedPath(dir->Str() + "/.data-a-0.parquet"), + dir->Str() + "/.data-a-0.parquet", {}, 1, 1); + ASSERT_NOK_WITH_MSG(commit->Commit({hidden_target}), "a scan of this table would skip"); + + // Two messages aiming at one path would have one silently overwrite the other. + ASSERT_OK(dir->GetFileSystem()->WriteFile(StagedPath(dir->Str() + "/data-a-0.parquet"), "x", + /*overwrite=*/true)); + FormatCommitMessage first(StagedPath(dir->Str() + "/data-a-0.parquet"), + dir->Str() + "/data-a-0.parquet", {}, 1, 1); + ASSERT_NOK_WITH_MSG(commit->Commit({first, first}), "would publish"); +} + +TEST(FormatTableTest, TestCommitChecksTheStagedFileBeforeTouchingAnything) { + std::unique_ptr dir = UniqueTestDirectory::Create(); + ASSERT_TRUE(dir); + ASSERT_OK_AND_ASSIGN(std::shared_ptr table, + CreateTable(dir->GetFileSystem(), dir->Str(), {})); + ASSERT_OK_AND_ASSIGN( + std::unique_ptr commit, + FormatTableCommit::Create(table, /*overwrite=*/true, /*static_partition=*/{})); + + // A message goes stale when its write was aborted, or when the same messages were committed + // once already. Nothing moves on the strength of a file that is not there. + FormatCommitMessage gone(StagedPath(dir->Str() + "/data-a-0.parquet"), + dir->Str() + "/data-a-0.parquet", {}, 1, 1); + ASSERT_NOK_WITH_MSG(commit->Commit({gone}), "cannot be read"); + + // `rename` moves a directory as readily as a file, and the overwrite has already cleared + // the old rows by then. + ASSERT_OK(dir->GetFileSystem()->Mkdirs(StagedPath(dir->Str() + "/data-b-0.parquet"))); + FormatCommitMessage a_directory(StagedPath(dir->Str() + "/data-b-0.parquet"), + dir->Str() + "/data-b-0.parquet", {}, 1, 0); + ASSERT_NOK_WITH_MSG(commit->Commit({a_directory}), "is a directory, not a file"); + + // A length that disagrees with the file means the message and the file are from different + // writes; publishing it would record a size nothing can rely on. + ASSERT_OK(dir->GetFileSystem()->WriteFile(StagedPath(dir->Str() + "/data-c-0.parquet"), "12345", + /*overwrite=*/true)); + FormatCommitMessage wrong_size(StagedPath(dir->Str() + "/data-c-0.parquet"), + dir->Str() + "/data-c-0.parquet", {}, 1, 999); + ASSERT_NOK_WITH_MSG(commit->Commit({wrong_size}), "but the commit message says"); + + // Nothing was published, and any old data was never cleared. + ASSERT_OK_AND_ASSIGN(bool published, + dir->GetFileSystem()->Exists(dir->Str() + "/data-c-0.parquet")); + ASSERT_FALSE(published); +} + +TEST(FormatTableTest, TestReadRefusesASplitOverInvisibleFiles) { + std::unique_ptr dir = UniqueTestDirectory::Create(); + ASSERT_TRUE(dir); + ASSERT_OK_AND_ASSIGN(std::shared_ptr table, + CreateTable(dir->GetFileSystem(), dir->Str(), {})); + ASSERT_OK_AND_ASSIGN( + std::unique_ptr read, + FormatTableRead::Create(table, /*projection=*/std::nullopt, /*pool=*/nullptr, + /*predicate=*/nullptr, + /*enable_predicate_filter=*/false)); + + // A scan skips a hidden name and never descends into one, since that is where an uncommitted + // job stages its output. A split that did not come from one must not read what a scan would + // never return. + auto staged = std::make_shared( + std::vector{{dir->Str() + "/_temporary/a.parquet", 1}}, + std::map{}); + ASSERT_NOK_WITH_MSG(read->CreateReader(std::static_pointer_cast(staged)), + "a scan of this table would skip"); + + // Nor this table's own metadata, which a file system catalog keeps under the location. + auto metadata = std::make_shared( + std::vector{{dir->Str() + "/schema/schema-0", 1}}, + std::map{}); + ASSERT_NOK_WITH_MSG(read->CreateReader(std::static_pointer_cast(metadata)), + "own metadata rather than data"); + + // A size no file could have is refused before anything is opened. + auto negative_size = std::make_shared( + std::vector{{dir->Str() + "/data-a-0.parquet", -1}}, + std::map{}); + ASSERT_NOK_WITH_MSG(read->CreateReader(std::static_pointer_cast(negative_size)), + "negative size"); +} + +TEST(FormatTableTest, TestAbortCleansUpTheGoodMessagesAmongBadOnes) { + std::unique_ptr dir = UniqueTestDirectory::Create(); + ASSERT_TRUE(dir); + ASSERT_OK_AND_ASSIGN(std::shared_ptr table, + CreateTable(dir->GetFileSystem(), dir->Str(), {})); + ASSERT_OK_AND_ASSIGN(std::unique_ptr batch, + MakeBatch({1}, {"alice"}, "20240101", {})); + ASSERT_OK_AND_ASSIGN(std::unique_ptr write, + FormatTableWrite::Create(table, /*pool=*/nullptr)); + ASSERT_OK(write->Write(std::move(batch))); + ASSERT_OK_AND_ASSIGN(std::vector messages, write->PrepareCommit()); + ASSERT_EQ(messages.size(), 1u); + + // Refusing the whole batch at the first bad message would leave the good ones' staged files + // behind with nothing left to clean them up. + std::vector mixed = { + FormatCommitMessage(StagedPath("/elsewhere/x"), "/elsewhere/x", {}, 1, 1), messages[0]}; + ASSERT_OK_AND_ASSIGN( + std::unique_ptr commit, + FormatTableCommit::Create(table, /*overwrite=*/false, /*static_partition=*/{})); + ASSERT_OK(commit->Abort(mixed)); + ASSERT_OK_AND_ASSIGN(bool temp_exists, + dir->GetFileSystem()->Exists(messages[0].temp_file_path)); + ASSERT_FALSE(temp_exists); +} + +TEST(FormatTableTest, TestReadRefusesASplitItCannotUse) { + std::unique_ptr dir = UniqueTestDirectory::Create(); + ASSERT_TRUE(dir); + + // `CreateReader` takes the base `Split`, which a managed table's splits are too. One of those + // would have this read looking for files where a format table keeps none, so the type is + // checked rather than assumed, as is the null a caller can always hand over. + class NotAFormatSplit : public Split {}; + ASSERT_OK_AND_ASSIGN(std::shared_ptr table, + CreateTable(dir->GetFileSystem(), dir->Str(), {"dt"})); + ASSERT_OK_AND_ASSIGN( + std::unique_ptr read, + FormatTableRead::Create(table, /*projection=*/std::nullopt, /*pool=*/nullptr, + /*predicate=*/nullptr, + /*enable_predicate_filter=*/false)); + ASSERT_NOK_WITH_MSG(read->CreateReader(std::make_shared()), + "only accepts a FormatDataSplit"); + ASSERT_NOK_WITH_MSG(read->CreateReader(std::shared_ptr()), + "only accepts a FormatDataSplit"); + + // A partition value the column type cannot hold. The directory name and the split agree, so + // nothing before this notices; only reading the value into its type does. A split is a public + // struct, so this has to be a refusal rather than an assumption. + std::unique_ptr int_dir = UniqueTestDirectory::Create(); + ASSERT_TRUE(int_dir); + auto int_schema = + arrow::schema({arrow::field("name", arrow::utf8()), arrow::field("pt", arrow::int32())}); + SchemaManager schema_manager(int_dir->GetFileSystem(), int_dir->Str()); + ASSERT_OK_AND_ASSIGN([[maybe_unused]] std::unique_ptr int_table_schema, + schema_manager.CreateTable( + int_schema, /*partition_keys=*/{"pt"}, /*primary_keys=*/{}, + {{Options::TYPE, "format-table"}, {Options::FILE_FORMAT, "parquet"}})); + ASSERT_OK_AND_ASSIGN( + std::shared_ptr int_table, + FormatTable::Create(int_dir->GetFileSystem(), int_dir->Str(), Identifier("db", "tbl"))); + ASSERT_OK_AND_ASSIGN( + std::unique_ptr int_read, + FormatTableRead::Create(int_table, /*projection=*/std::nullopt, /*pool=*/nullptr, + /*predicate=*/nullptr, + /*enable_predicate_filter=*/false)); + auto not_an_int = std::make_shared( + std::vector{{int_dir->Str() + "/pt=abc/data-a-0.parquet", 1}}, + std::map{{"pt", "abc"}}); + ASSERT_NOK(int_read->CreateReader(std::static_pointer_cast(not_an_int))); +} + +TEST(FormatTableTest, TestReadRefusesASplitFromOutsideTheTable) { + std::unique_ptr dir = UniqueTestDirectory::Create(); + ASSERT_TRUE(dir); + ASSERT_OK_AND_ASSIGN(std::shared_ptr table, + CreateTable(dir->GetFileSystem(), dir->Str(), {"dt"})); + ASSERT_OK_AND_ASSIGN( + std::unique_ptr read, + FormatTableRead::Create(table, /*projection=*/std::nullopt, /*pool=*/nullptr, + /*predicate=*/nullptr, + /*enable_predicate_filter=*/false)); + + // Every file a split names is opened, and the split reaching `CreateReader()` may have been + // planned from somewhere else entirely. Whether a file belongs to this table is a question + // only the table can answer. + auto outside = std::make_shared( + std::vector{{"/etc/passwd", 1}}, + std::map{{"dt", "20240101"}}); + ASSERT_NOK_WITH_MSG(read->CreateReader(std::static_pointer_cast(outside)), + "not under the table location"); + + auto escaping = std::make_shared( + std::vector{{dir->Str() + "/../victim/a.parquet", 1}}, + std::map{{"dt", "20240101"}}); + ASSERT_NOK_WITH_MSG(read->CreateReader(std::static_pointer_cast(escaping)), + "does not stay inside"); + + // A file from another partition would read back under partition values it never had. + auto wrong_partition = std::make_shared( + std::vector{{dir->Str() + "/dt=20240102/a.parquet", 1}}, + std::map{{"dt", "20240101"}}); + ASSERT_NOK_WITH_MSG(read->CreateReader(std::static_pointer_cast(wrong_partition)), + "but claims"); + + // And a split that names no partition at all cannot say where its rows belong. + auto no_partition = std::make_shared( + std::vector{{dir->Str() + "/dt=20240101/a.parquet", 1}}, + std::map{}); + ASSERT_NOK_WITH_MSG(read->CreateReader(std::static_pointer_cast(no_partition)), + "partition values"); +} + +TEST(FormatTableTest, TestAZeroRowBatchWritesNoFile) { + std::unique_ptr dir = UniqueTestDirectory::Create(); + ASSERT_TRUE(dir); + // A writer is created when the first row arrives, so a write that receives none leaves the + // directory as it found it, rather than a file holding nothing but a footer. + ASSERT_OK_AND_ASSIGN(std::shared_ptr table, + CreateTable(dir->GetFileSystem(), dir->Str(), {})); + ASSERT_OK_AND_ASSIGN(std::unique_ptr empty, MakeBatch({}, {}, "20240101", {})); + ASSERT_OK_AND_ASSIGN(std::unique_ptr write, + FormatTableWrite::Create(table, /*pool=*/nullptr)); + ASSERT_OK(write->Write(std::move(empty))); + ASSERT_OK_AND_ASSIGN(std::vector messages, write->PrepareCommit()); + ASSERT_TRUE(messages.empty()); +} + +TEST(FormatTableTest, TestAbortRemovesWrittenFiles) { + std::unique_ptr dir = UniqueTestDirectory::Create(); + ASSERT_TRUE(dir); + ASSERT_OK_AND_ASSIGN(std::shared_ptr table, + CreateTable(dir->GetFileSystem(), dir->Str(), {})); + ASSERT_OK_AND_ASSIGN(std::unique_ptr batch, + MakeBatch({1}, {"alice"}, "20240101", {})); + ASSERT_OK_AND_ASSIGN(std::unique_ptr write, + FormatTableWrite::Create(table, /*pool=*/nullptr)); + ASSERT_OK(write->Write(std::move(batch))); + ASSERT_OK_AND_ASSIGN(std::vector messages, write->PrepareCommit()); + + ASSERT_OK_AND_ASSIGN( + std::unique_ptr commit, + FormatTableCommit::Create(table, /*overwrite=*/false, /*static_partition=*/{})); + ASSERT_OK(commit->Abort(messages)); + ASSERT_OK_AND_ASSIGN(bool temp_exists, + dir->GetFileSystem()->Exists(messages[0].temp_file_path)); + ASSERT_FALSE(temp_exists); + ASSERT_OK_AND_ASSIGN(bool file_exists, dir->GetFileSystem()->Exists(messages[0].file_path)); + ASSERT_FALSE(file_exists); +} + +TEST(FormatTableTest, TestWriteRejectsNonInsertRows) { + std::unique_ptr dir = UniqueTestDirectory::Create(); + ASSERT_TRUE(dir); + ASSERT_OK_AND_ASSIGN(std::shared_ptr table, + CreateTable(dir->GetFileSystem(), dir->Str(), {})); + ASSERT_OK_AND_ASSIGN(std::unique_ptr delete_batch, + MakeBatch({1}, {"alice"}, "20240101", {}, {RecordBatch::RowKind::DELETE})); + + ASSERT_OK_AND_ASSIGN(std::unique_ptr write, + FormatTableWrite::Create(table, /*pool=*/nullptr)); + Status status = write->Write(std::move(delete_batch)); + ASSERT_FALSE(status.ok()); + ASSERT_TRUE(status.IsInvalid()); + ASSERT_OK(write->Abort()); +} + +TEST(FormatTableTest, TestWriteRejectsRowsFromAnotherPartition) { + std::unique_ptr dir = UniqueTestDirectory::Create(); + ASSERT_TRUE(dir); + ASSERT_OK_AND_ASSIGN(std::shared_ptr table, + CreateTable(dir->GetFileSystem(), dir->Str(), {"dt"})); + // The rows say they belong to 20240102 while the batch declares 20240101. The partition + // column is not written, so accepting this would file the rows under the wrong partition and + // lose their real value. + ASSERT_OK_AND_ASSIGN(std::unique_ptr batch, + MakeBatch({1}, {"alice"}, "20240102", {{"dt", "20240101"}})); + ASSERT_OK_AND_ASSIGN(std::unique_ptr write, + FormatTableWrite::Create(table, /*pool=*/nullptr)); + Status status = write->Write(std::move(batch)); + ASSERT_FALSE(status.ok()); + ASSERT_TRUE(status.IsInvalid()); + ASSERT_OK(write->Abort()); +} + +TEST(FormatTableTest, TestASplitWeighsTheFilesItNames) { + // What a split weighs decides how the scan packs it, so it weighs the files it names. + FormatDataSplit split({{"/tbl/data-a-0.parquet", 4096}, {"/tbl/data-a-1.parquet", 512}}, {}); + ASSERT_EQ(split.files.size(), 2u); + ASSERT_EQ(split.files[0].file_size, 4096); + ASSERT_EQ(split.TotalSize(), 4608); + + // The sizes are whatever the split was given, so a total that would not fit an int64 + // saturates rather than wrapping into a negative answer, which the scan would then pack as + // if the files were empty. + constexpr int64_t kMax = std::numeric_limits::max(); + FormatDataSplit huge({{"/tbl/a.parquet", kMax}, {"/tbl/b.parquet", kMax}}, {}); + ASSERT_EQ(huge.TotalSize(), kMax); +} + +TEST(FormatTableTest, TestASplitIsNotSerializable) { + // A format table's plan has no cross-runtime encoding, and one of paimon-cpp's own would let + // a plan made here be handed to a runtime that cannot read it back. + auto split = std::make_shared( + std::vector{{"/tbl/dt=20240101/data-a-0.parquet", 128}}, + std::map{{"dt", "20240101"}}); + Result serialized = Split::Serialize(split, GetDefaultPool()); + ASSERT_FALSE(serialized.ok()); + ASSERT_TRUE(serialized.status().IsNotImplemented()); + ASSERT_NE(serialized.status().message().find("in-memory only"), std::string::npos); +} + +TEST(FormatTableTest, TestACommitMessageIsNotSerializable) { + // The other half of the same rule: a message names a staged path rather than files to record + // in a manifest, so there is nothing a manifest-shaped encoding could carry. + auto message = std::make_shared( + StagedPath("/tbl/dt=20240101/data-a-0.parquet"), "/tbl/dt=20240101/data-a-0.parquet", + std::map{{"dt", "20240101"}}, /*record_count=*/1, + /*file_size=*/128); + Result serialized = CommitMessage::Serialize(message, GetDefaultPool()); + ASSERT_FALSE(serialized.ok()); + ASSERT_TRUE(serialized.status().IsNotImplemented()); + ASSERT_NE(serialized.status().message().find("belong to one process"), std::string::npos); + + // And through the list form, which is what a sink hands a batch of messages to. + Result serialized_list = CommitMessage::SerializeList( + std::vector>{message}, GetDefaultPool()); + ASSERT_FALSE(serialized_list.ok()); + ASSERT_TRUE(serialized_list.status().IsNotImplemented()); +} + +TEST(FormatTableTest, TestOverwriteReplacesTheWholePartitionOfANestedFile) { + // A commit message may name a file below its partition directory, which is what lets a + // partition keep its files in plain subdirectories. An overwrite of such a message replaces + // everything the partition holds, not just the subdirectory the new file lands in: the + // partition is what the overwrite was asked to replace. + std::unique_ptr dir = UniqueTestDirectory::Create(); + ASSERT_TRUE(dir); + ASSERT_OK_AND_ASSIGN(std::shared_ptr table, + CreateTable(dir->GetFileSystem(), dir->Str(), {"dt"})); + const std::string partition_dir = PathUtil::JoinPath(dir->Str(), "dt=20240101"); + + // Old data at the partition root, and a copy of it in a sibling subdirectory of the partition. + ASSERT_OK_AND_ASSIGN(std::unique_ptr old_batch, + MakeBatch({1, 2}, {"alice", "bob"}, "20240101", {{"dt", "20240101"}})); + ASSERT_OK(WriteAndCommit(table, std::move(old_batch))); + ASSERT_OK_AND_ASSIGN( + std::unique_ptr before, + FormatTableScan::Create(table, /*partition_filter=*/{}, /*limit=*/std::nullopt)); + ASSERT_OK_AND_ASSIGN(std::shared_ptr before_plan, before->CreatePlan()); + ASSERT_EQ(before_plan->Splits().size(), 1u); + auto before_split = std::dynamic_pointer_cast(before_plan->Splits()[0]); + ASSERT_NE(before_split, nullptr); + ASSERT_EQ(before_split->files.size(), 1u); + const std::string old_root_file = before_split->files[0].file_path; + std::string old_content; + ASSERT_OK(dir->GetFileSystem()->ReadFile(old_root_file, &old_content)); + const std::string sibling_dir = PathUtil::JoinPath(partition_dir, "part-1"); + ASSERT_OK(dir->GetFileSystem()->Mkdirs(sibling_dir)); + ASSERT_OK(dir->GetFileSystem()->WriteFile( + PathUtil::JoinPath(sibling_dir, PathUtil::GetName(old_root_file)), old_content, + /*overwrite=*/true)); + + // A new file staged for a subdirectory of the same partition. The bytes are a real data file, + // written through the table and then staged where a message may legitimately name one. + ASSERT_OK_AND_ASSIGN(std::unique_ptr write, + FormatTableWrite::Create(table, /*pool=*/nullptr)); + ASSERT_OK_AND_ASSIGN(std::unique_ptr new_batch, + MakeBatch({3}, {"carol"}, "20240101", {{"dt", "20240101"}})); + ASSERT_OK(write->Write(std::move(new_batch))); + ASSERT_OK_AND_ASSIGN(std::vector written, write->PrepareCommit()); + ASSERT_EQ(written.size(), 1u); + std::string new_content; + ASSERT_OK(dir->GetFileSystem()->ReadFile(written[0].temp_file_path, &new_content)); + ASSERT_OK(write->Abort()); + + const std::string nested_target = + PathUtil::JoinPath(PathUtil::JoinPath(partition_dir, "part-0"), "data-nested-0.parquet"); + const std::string nested_staged = StagedPath(nested_target); + ASSERT_OK(dir->GetFileSystem()->Mkdirs(PathUtil::GetParentDirPath(nested_staged))); + ASSERT_OK(dir->GetFileSystem()->WriteFile(nested_staged, new_content, /*overwrite=*/true)); + ASSERT_OK_AND_ASSIGN(FileStatus staged_status, + dir->GetFileSystem()->GetFileStatus(nested_staged)); + + ASSERT_OK_AND_ASSIGN( + std::unique_ptr commit, + FormatTableCommit::Create(table, /*overwrite=*/true, /*static_partition=*/{})); + FormatCommitMessage nested(nested_staged, nested_target, + std::map{{"dt", "20240101"}}, + written[0].record_count, staged_status.GetLen()); + ASSERT_OK(commit->Commit({nested})); + + // Only the new row is left: the old file at the partition root and the one in the sibling + // subdirectory were both replaced. + ASSERT_OK_AND_ASSIGN( + std::unique_ptr after, + FormatTableScan::Create(table, /*partition_filter=*/{}, /*limit=*/std::nullopt)); + ASSERT_OK_AND_ASSIGN(std::shared_ptr after_plan, after->CreatePlan()); + ASSERT_OK_AND_ASSIGN(std::vector rows, ReadAll(table, after_plan->Splits())); + ASSERT_EQ(rows, (std::vector{"3|carol|20240101"})); +} + +TEST(FormatTableTest, TestOverwriteReplacesThePartitionsItWrites) { + std::unique_ptr dir = UniqueTestDirectory::Create(); + ASSERT_TRUE(dir); + ASSERT_OK_AND_ASSIGN(std::shared_ptr table, + CreateTable(dir->GetFileSystem(), dir->Str(), {"dt"})); + ASSERT_OK_AND_ASSIGN(std::unique_ptr first, + MakeBatch({1, 2}, {"alice", "bob"}, "20240101", {{"dt", "20240101"}})); + ASSERT_OK(WriteAndCommit(table, std::move(first))); + ASSERT_OK_AND_ASSIGN(std::unique_ptr other, + MakeBatch({9}, {"zoe"}, "20240102", {{"dt", "20240102"}})); + ASSERT_OK(WriteAndCommit(table, std::move(other))); + + ASSERT_OK_AND_ASSIGN(std::unique_ptr replacement, + MakeBatch({3}, {"carol"}, "20240101", {{"dt", "20240101"}})); + ASSERT_OK(WriteAndCommit(table, std::move(replacement), /*overwrite=*/true)); + + ASSERT_OK_AND_ASSIGN( + std::unique_ptr scan, + FormatTableScan::Create(table, /*partition_filter=*/{}, /*limit=*/std::nullopt)); + ASSERT_OK_AND_ASSIGN(std::shared_ptr plan, scan->CreatePlan()); + ASSERT_OK_AND_ASSIGN(std::vector rows, ReadAll(table, plan->Splits())); + std::sort(rows.begin(), rows.end()); + // Only the partition written to is replaced; the one this commit never touched is left alone. + ASSERT_EQ(rows, (std::vector{"3|carol|20240101", "9|zoe|20240102"})); +} + +TEST(FormatTableTest, TestOverwriteWithAStaticPartitionClearsThatPartition) { + std::unique_ptr dir = UniqueTestDirectory::Create(); + ASSERT_TRUE(dir); + ASSERT_OK_AND_ASSIGN(std::shared_ptr table, + CreateTable(dir->GetFileSystem(), dir->Str(), {"dt"})); + ASSERT_OK_AND_ASSIGN(std::unique_ptr first, + MakeBatch({1, 2}, {"alice", "bob"}, "20240101", {{"dt", "20240101"}})); + ASSERT_OK(WriteAndCommit(table, std::move(first))); + ASSERT_OK_AND_ASSIGN(std::unique_ptr other, + MakeBatch({9}, {"zoe"}, "20240102", {{"dt", "20240102"}})); + ASSERT_OK(WriteAndCommit(table, std::move(other))); + + ASSERT_OK_AND_ASSIGN(std::unique_ptr replacement, + MakeBatch({3}, {"carol"}, "20240101", {{"dt", "20240101"}})); + ASSERT_OK( + WriteAndCommit(table, std::move(replacement), /*overwrite=*/true, {{"dt", "20240101"}})); + + ASSERT_OK_AND_ASSIGN( + std::unique_ptr scan, + FormatTableScan::Create(table, /*partition_filter=*/{}, /*limit=*/std::nullopt)); + ASSERT_OK_AND_ASSIGN(std::shared_ptr plan, scan->CreatePlan()); + ASSERT_OK_AND_ASSIGN(std::vector rows, ReadAll(table, plan->Splits())); + std::sort(rows.begin(), rows.end()); + ASSERT_EQ(rows, (std::vector{"3|carol|20240101", "9|zoe|20240102"})); +} + +TEST(FormatTableTest, TestOverwriteOfAPartitionThatIsNotThereYetSucceeds) { + // Overwriting a partition that does not exist has nothing to clear, and is how a first write + // to that partition is spelled. Failing on it would make an overwriting job depend on whether + // some earlier job had already created the directory. + std::unique_ptr dir = UniqueTestDirectory::Create(); + ASSERT_TRUE(dir); + ASSERT_OK_AND_ASSIGN(std::shared_ptr table, + CreateTable(dir->GetFileSystem(), dir->Str(), {"dt"})); + + ASSERT_OK_AND_ASSIGN(std::unique_ptr batch, + MakeBatch({1}, {"alice"}, "20240101", {{"dt", "20240101"}})); + ASSERT_OK(WriteAndCommit(table, std::move(batch), /*overwrite=*/true, {{"dt", "20240101"}})); + + ASSERT_OK_AND_ASSIGN( + std::unique_ptr scan, + FormatTableScan::Create(table, /*partition_filter=*/{}, /*limit=*/std::nullopt)); + ASSERT_OK_AND_ASSIGN(std::shared_ptr plan, scan->CreatePlan()); + ASSERT_OK_AND_ASSIGN(std::vector rows, ReadAll(table, plan->Splits())); + ASSERT_EQ(rows, (std::vector{"1|alice|20240101"})); + + // And with nothing to publish either: the partition is left behind empty rather than not + // created at all, so a later scan lists it. + ASSERT_OK_AND_ASSIGN( + std::unique_ptr commit, + FormatTableCommit::Create(table, /*overwrite=*/true, {{"dt", "20240202"}})); + ASSERT_OK(commit->Commit({})); + ASSERT_OK_AND_ASSIGN( + std::unique_ptr after, + FormatTableScan::Create(table, /*partition_filter=*/{}, /*limit=*/std::nullopt)); + ASSERT_OK_AND_ASSIGN(PartitionList partitions, after->ListPartitions()); + ASSERT_EQ(partitions.size(), 2u); + ASSERT_EQ(partitions[1].at("dt"), "20240202"); +} + +TEST(FormatTableTest, TestOverwriteCanEmptyAPartitionWithoutWritingToIt) { + std::unique_ptr dir = UniqueTestDirectory::Create(); + ASSERT_TRUE(dir); + ASSERT_OK_AND_ASSIGN(std::shared_ptr table, + CreateTable(dir->GetFileSystem(), dir->Str(), {"dt"})); + ASSERT_OK_AND_ASSIGN(std::unique_ptr first, + MakeBatch({1, 2}, {"alice", "bob"}, "20240101", {{"dt", "20240101"}})); + ASSERT_OK(WriteAndCommit(table, std::move(first))); + + // An overwrite that names a partition but writes nothing into it clears that partition and + // leaves it behind empty, rather than removing it from the table. + ASSERT_OK_AND_ASSIGN( + std::unique_ptr commit, + FormatTableCommit::Create(table, /*overwrite=*/true, {{"dt", "20240101"}})); + ASSERT_OK(commit->Commit({})); + + ASSERT_OK_AND_ASSIGN( + std::unique_ptr scan, + FormatTableScan::Create(table, /*partition_filter=*/{}, /*limit=*/std::nullopt)); + ASSERT_OK_AND_ASSIGN(std::shared_ptr plan, scan->CreatePlan()); + ASSERT_OK_AND_ASSIGN(std::vector rows, ReadAll(table, plan->Splits())); + ASSERT_TRUE(rows.empty()); + // The partition itself is still there, with no rows in it. + ASSERT_OK_AND_ASSIGN(PartitionList partitions, scan->ListPartitions()); + ASSERT_EQ(partitions.size(), 1u); + ASSERT_EQ(partitions[0], (std::map{{"dt", "20240101"}})); +} + +TEST(FormatTableTest, TestDataFilePrefixReachesTheWrittenFiles) { + std::unique_ptr dir = UniqueTestDirectory::Create(); + ASSERT_TRUE(dir); + ASSERT_OK_AND_ASSIGN( + std::shared_ptr table, + CreateTable(dir->GetFileSystem(), dir->Str(), {}, {{Options::DATA_FILE_PREFIX, "part-"}})); + ASSERT_OK_AND_ASSIGN(std::unique_ptr write, + FormatTableWrite::Create(table, /*pool=*/nullptr)); + ASSERT_OK_AND_ASSIGN(std::unique_ptr batch, + MakeBatch({1}, {"alice"}, "20240101", {})); + ASSERT_OK(write->Write(std::move(batch))); + ASSERT_OK_AND_ASSIGN(std::vector messages, write->PrepareCommit()); + ASSERT_EQ(messages.size(), 1u); + ASSERT_TRUE(StringUtils::StartsWith(PathUtil::GetName(messages[0].file_path), "part-")) + << messages[0].file_path; + ASSERT_OK(write->Abort()); +} + +TEST(FormatTableTest, TestCommitWithoutOverwriteAddsToWhatIsThere) { + std::unique_ptr dir = UniqueTestDirectory::Create(); + ASSERT_TRUE(dir); + ASSERT_OK_AND_ASSIGN(std::shared_ptr table, + CreateTable(dir->GetFileSystem(), dir->Str(), {"dt"})); + ASSERT_OK_AND_ASSIGN(std::unique_ptr first, + MakeBatch({1}, {"alice"}, "20240101", {{"dt", "20240101"}})); + ASSERT_OK(WriteAndCommit(table, std::move(first))); + ASSERT_OK_AND_ASSIGN(std::unique_ptr second, + MakeBatch({2}, {"bob"}, "20240101", {{"dt", "20240101"}})); + ASSERT_OK(WriteAndCommit(table, std::move(second))); + + ASSERT_OK_AND_ASSIGN( + std::unique_ptr scan, + FormatTableScan::Create(table, /*partition_filter=*/{}, /*limit=*/std::nullopt)); + ASSERT_OK_AND_ASSIGN(std::shared_ptr plan, scan->CreatePlan()); + ASSERT_OK_AND_ASSIGN(std::vector rows, ReadAll(table, plan->Splits())); + std::sort(rows.begin(), rows.end()); + ASSERT_EQ(rows, (std::vector{"1|alice|20240101", "2|bob|20240101"})); +} + +TEST(FormatTableTest, TestStaticPartitionMustNameLeadingKeysOfAPartitionedTable) { + std::unique_ptr dir = UniqueTestDirectory::Create(); + ASSERT_TRUE(dir); + ASSERT_OK_AND_ASSIGN(std::shared_ptr table, + CreateTable(dir->GetFileSystem(), dir->Str(), {"dt"})); + ASSERT_NOK_WITH_MSG(FormatTableCommit::Create(table, /*overwrite=*/true, {{"nope", "1"}}), + "is not a partition key"); + + ASSERT_OK_AND_ASSIGN(std::shared_ptr unpartitioned, + CreateTable(dir->GetFileSystem(), dir->Str() + "/plain", {})); + ASSERT_NOK_WITH_MSG( + FormatTableCommit::Create(unpartitioned, /*overwrite=*/true, {{"dt", "20240101"}}), + "is not partitioned"); +} + +TEST(FormatTableTest, TestReadRefusesASplitWhoseFileChanged) { + std::unique_ptr dir = UniqueTestDirectory::Create(); + ASSERT_TRUE(dir); + ASSERT_OK_AND_ASSIGN(std::shared_ptr table, + CreateTable(dir->GetFileSystem(), dir->Str(), {})); + ASSERT_OK(dir->GetFileSystem()->WriteFile(dir->Str() + "/data-x-0.parquet", "not a data file", + /*overwrite=*/true)); + ASSERT_OK_AND_ASSIGN( + std::unique_ptr read, + FormatTableRead::Create(table, /*projection=*/std::nullopt, /*pool=*/nullptr, + /*predicate=*/nullptr, + /*enable_predicate_filter=*/false)); + + // The size in a split is the caller's and `Open` trusts it, so a stale length would truncate + // an object-store read. A file is opened only when it is reached, so the check runs then + // rather than when the reader is built. + auto wrong_size = std::make_shared( + std::vector{{dir->Str() + "/data-x-0.parquet", 999999}}, + std::map{}); + ASSERT_OK_AND_ASSIGN(std::unique_ptr wrong_size_reader, + read->CreateReader(std::static_pointer_cast(wrong_size))); + Result stale = wrong_size_reader->NextBatch(); + ASSERT_NOK_WITH_MSG(stale, "different version of the file"); + // And it says which file, since a split holds a whole partition of them. + ASSERT_NOK_WITH_MSG(stale, "data-x-0.parquet"); + wrong_size_reader->Close(); + + ASSERT_OK(dir->GetFileSystem()->Mkdirs(dir->Str() + "/adir")); + auto a_directory = std::make_shared( + std::vector{{dir->Str() + "/adir", 0}}, + std::map{}); + ASSERT_OK_AND_ASSIGN(std::unique_ptr directory_reader, + read->CreateReader(std::static_pointer_cast(a_directory))); + Result not_a_file = directory_reader->NextBatch(); + ASSERT_NOK_WITH_MSG(not_a_file, "names a directory, not a data file"); + ASSERT_NOK_WITH_MSG(not_a_file, "adir"); + directory_reader->Close(); +} + +TEST(FormatTableTest, TestPredicateFilterKeepsOnlyMatchingRows) { + std::unique_ptr dir = UniqueTestDirectory::Create(); + ASSERT_TRUE(dir); + ASSERT_OK_AND_ASSIGN(std::shared_ptr table, + CreateTable(dir->GetFileSystem(), dir->Str(), {"dt"})); + ASSERT_OK_AND_ASSIGN( + std::unique_ptr batch, + MakeBatch({1, 2, 3}, {"alice", "bob", "carol"}, "20240101", {{"dt", "20240101"}})); + ASSERT_OK(WriteAndCommit(table, std::move(batch))); + + ASSERT_OK_AND_ASSIGN( + std::unique_ptr scan, + FormatTableScan::Create(table, /*partition_filter=*/{}, /*limit=*/std::nullopt)); + ASSERT_OK_AND_ASSIGN(std::shared_ptr plan, scan->CreatePlan()); + std::shared_ptr id_gt_1 = PredicateBuilder::GreaterThan( + /*field_index=*/0, /*field_name=*/"id", FieldType::INT, Literal(1)); + + ASSERT_OK_AND_ASSIGN(std::vector rows, ReadAll(table, plan->Splits(), id_gt_1, + /*enable_predicate_filter=*/true)); + ASSERT_EQ(rows, (std::vector{"2|bob|20240101", "3|carol|20240101"})); +} + +TEST(FormatTableTest, TestPredicateWithoutFilterIsOnlyPushedDown) { + std::unique_ptr dir = UniqueTestDirectory::Create(); + ASSERT_TRUE(dir); + ASSERT_OK_AND_ASSIGN(std::shared_ptr table, + CreateTable(dir->GetFileSystem(), dir->Str(), {"dt"})); + ASSERT_OK_AND_ASSIGN( + std::unique_ptr batch, + MakeBatch({1, 2, 3}, {"alice", "bob", "carol"}, "20240101", {{"dt", "20240101"}})); + ASSERT_OK(WriteAndCommit(table, std::move(batch))); + + ASSERT_OK_AND_ASSIGN( + std::unique_ptr scan, + FormatTableScan::Create(table, /*partition_filter=*/{}, /*limit=*/std::nullopt)); + ASSERT_OK_AND_ASSIGN(std::shared_ptr plan, scan->CreatePlan()); + std::shared_ptr id_gt_1 = PredicateBuilder::GreaterThan( + /*field_index=*/0, /*field_name=*/"id", FieldType::INT, Literal(1)); + + // A one-sided contract: what the predicate keeps is never lost, what it rejects may still + // come back. Asserting a row count would assert the format's statistics granularity. + ASSERT_OK_AND_ASSIGN(std::vector rows, ReadAll(table, plan->Splits(), id_gt_1)); + ASSERT_NE(std::find(rows.begin(), rows.end(), "2|bob|20240101"), rows.end()); + ASSERT_NE(std::find(rows.begin(), rows.end(), "3|carol|20240101"), rows.end()); + ASSERT_LE(rows.size(), 3u); + + // Exactness is what `enable_predicate_filter` is for, and then the rejected row is gone. + ASSERT_OK_AND_ASSIGN(std::vector filtered, + ReadAll(table, plan->Splits(), id_gt_1, + /*enable_predicate_filter=*/true)); + ASSERT_EQ(filtered, (std::vector{"2|bob|20240101", "3|carol|20240101"})); +} + +TEST(FormatTableTest, TestPredicateOnPartitionColumnIsAppliedToTheBatch) { + std::unique_ptr dir = UniqueTestDirectory::Create(); + ASSERT_TRUE(dir); + ASSERT_OK_AND_ASSIGN(std::shared_ptr table, + CreateTable(dir->GetFileSystem(), dir->Str(), {"dt"})); + ASSERT_OK_AND_ASSIGN(std::unique_ptr first, + MakeBatch({1}, {"alice"}, "20240101", {{"dt", "20240101"}})); + ASSERT_OK(WriteAndCommit(table, std::move(first))); + ASSERT_OK_AND_ASSIGN(std::unique_ptr second, + MakeBatch({2}, {"bob"}, "20240102", {{"dt", "20240102"}})); + ASSERT_OK(WriteAndCommit(table, std::move(second))); + + ASSERT_OK_AND_ASSIGN( + std::unique_ptr scan, + FormatTableScan::Create(table, /*partition_filter=*/{}, /*limit=*/std::nullopt)); + ASSERT_OK_AND_ASSIGN(std::shared_ptr plan, scan->CreatePlan()); + // The column lives in the directory name, not in the file, so only the batch can be tested + // against it, which is what the filter layer does. + const std::string wanted_dt = "20240102"; + std::shared_ptr dt_equal = PredicateBuilder::Equal( + /*field_index=*/2, /*field_name=*/"dt", FieldType::STRING, + Literal(FieldType::STRING, wanted_dt.data(), wanted_dt.size(), /*own_data=*/true)); + ASSERT_OK_AND_ASSIGN(std::vector rows, ReadAll(table, plan->Splits(), dt_equal, + /*enable_predicate_filter=*/true)); + ASSERT_EQ(rows, (std::vector{"2|bob|20240102"})); +} + +TEST(FormatTableTest, TestProjectionIsCheckedAgainstTheTable) { + std::unique_ptr dir = UniqueTestDirectory::Create(); + ASSERT_TRUE(dir); + ASSERT_OK_AND_ASSIGN(std::shared_ptr table, + CreateTable(dir->GetFileSystem(), dir->Str(), {"dt"})); + + // Reading the column once and naming the result twice would collide in the arrow schema, so + // a repeated column is refused rather than silently returned once. + std::vector projection = {"id", "name", "id"}; + ASSERT_NOK_WITH_MSG( + FormatTableRead::Create(table, projection, /*pool=*/nullptr, /*predicate=*/nullptr, + /*enable_predicate_filter=*/false), + "appears more than once in the projection"); + + // A partition column is read from the directory rather than from the file, and repeats the + // same way. + std::vector repeated_partition = {"dt", "dt"}; + ASSERT_NOK_WITH_MSG( + FormatTableRead::Create(table, repeated_partition, /*pool=*/nullptr, /*predicate=*/nullptr, + /*enable_predicate_filter=*/false), + "appears more than once in the projection"); + + // A column the table does not have has nothing to read: the projection is refused where it is + // given rather than coming back as a batch quietly missing a column. + ASSERT_NOK_WITH_MSG( + FormatTableRead::Create(table, std::vector{"id", "nosuchcolumn"}, + /*pool=*/nullptr, /*predicate=*/nullptr, + /*enable_predicate_filter=*/false), + "is not a column of table"); + + // And a projection has to name something. + ASSERT_NOK_WITH_MSG(FormatTableRead::Create(table, std::vector{}, + /*pool=*/nullptr, /*predicate=*/nullptr, + /*enable_predicate_filter=*/false), + "requires at least one column to read"); +} + +TEST(FormatTableTest, TestPredicateIsCheckedTheWayEveryOtherReadPathChecksIt) { + std::unique_ptr dir = UniqueTestDirectory::Create(); + ASSERT_TRUE(dir); + ASSERT_OK_AND_ASSIGN(std::shared_ptr table, + CreateTable(dir->GetFileSystem(), dir->Str(), {"dt"})); + + // The same two checks `TableRead` runs, through the same validator: a caller that moves a + // predicate between the managed and the format table path should not find one of them + // accepting what the other refuses. + std::shared_ptr literal_mismatch = PredicateBuilder::Equal( + /*field_index=*/0, /*field_name=*/"id", FieldType::INT, Literal(int64_t{1})); + ASSERT_NOK_WITH_MSG(FormatTableRead::Create(table, /*projection=*/std::nullopt, + /*pool=*/nullptr, literal_mismatch, + /*enable_predicate_filter=*/false), + "mismatch field type"); + + std::shared_ptr schema_mismatch = PredicateBuilder::Equal( + /*field_index=*/0, /*field_name=*/"id", FieldType::BIGINT, Literal(int64_t{1})); + Result> wrong_type = + FormatTableRead::Create(table, /*projection=*/std::nullopt, + /*pool=*/nullptr, schema_mismatch, + /*enable_predicate_filter=*/false); + ASSERT_FALSE(wrong_type.ok()); + ASSERT_TRUE(wrong_type.status().IsInvalid()) << wrong_type.status().ToString(); + + // The field index a predicate carries is not one of the checks: everything downstream + // resolves a field by name, and a caller that built the predicate against the table cannot + // know where a projection will put the column. + std::shared_ptr wrong_index = PredicateBuilder::Equal( + /*field_index=*/7, /*field_name=*/"id", FieldType::INT, Literal(1)); + ASSERT_OK(FormatTableRead::Create(table, /*projection=*/std::nullopt, + /*pool=*/nullptr, wrong_index, + /*enable_predicate_filter=*/false)); +} + +TEST(FormatTableTest, TestPredicateMayOnlyNameColumnsTheReadProduces) { + // One rule, the one `InternalReadContext` applies to a managed table: validated against the + // read schema. An unknown field and a dropped one are refused alike, since nothing + // downstream could evaluate either. + std::unique_ptr dir = UniqueTestDirectory::Create(); + ASSERT_TRUE(dir); + ASSERT_OK_AND_ASSIGN(std::shared_ptr table, + CreateTable(dir->GetFileSystem(), dir->Str(), {"dt"})); + + // Not a column of the table at all. + std::shared_ptr unknown = PredicateBuilder::Equal( + /*field_index=*/0, /*field_name=*/"nope", FieldType::INT, Literal(1)); + ASSERT_NOK_WITH_MSG( + FormatTableRead::Create(table, /*projection=*/std::nullopt, /*pool=*/nullptr, unknown, + /*enable_predicate_filter=*/false), + "does not exist in schema"); + + // A column of the table that this read does not produce. + std::shared_ptr id_gt_1 = PredicateBuilder::GreaterThan( + /*field_index=*/0, /*field_name=*/"id", FieldType::INT, Literal(1)); + std::vector projection = {"name", "dt"}; + ASSERT_NOK_WITH_MSG(FormatTableRead::Create(table, projection, /*pool=*/nullptr, id_gt_1, + /*enable_predicate_filter=*/true), + "does not exist in schema"); + ASSERT_NOK(FormatTableRead::Create(table, projection, /*pool=*/nullptr, id_gt_1, + /*enable_predicate_filter=*/false)); + + // Projected back in, and the same predicate is accepted. + ASSERT_OK(FormatTableRead::Create(table, std::vector{"id", "name", "dt"}, + /*pool=*/nullptr, id_gt_1, + /*enable_predicate_filter=*/true)); +} + +TEST(FormatTableTest, TestAPartitionValueIsReadIntoItsColumnTypeBeforeItNamesADirectory) { + // The declared partition is text, but the directory is named after the value it stands for: + // read into the column's type and rendered back, as Java Paimon's writer does. So two + // spellings of one value are one partition, not two half-filled directories. + std::unique_ptr dir = UniqueTestDirectory::Create(); + ASSERT_TRUE(dir); + auto int_schema = + arrow::schema({arrow::field("name", arrow::utf8()), arrow::field("pt", arrow::int32())}); + SchemaManager schema_manager(dir->GetFileSystem(), dir->Str()); + ASSERT_OK_AND_ASSIGN([[maybe_unused]] std::unique_ptr table_schema, + schema_manager.CreateTable( + int_schema, /*partition_keys=*/{"pt"}, /*primary_keys=*/{}, + {{Options::TYPE, "format-table"}, {Options::FILE_FORMAT, "parquet"}})); + ASSERT_OK_AND_ASSIGN( + std::shared_ptr table, + FormatTable::Create(dir->GetFileSystem(), dir->Str(), Identifier("db", "tbl"))); + + auto make_batch = [&int_schema]( + const std::string& name, + const std::string& declared) -> Result> { + arrow::StringBuilder name_builder; + arrow::Int32Builder pt_builder; + PAIMON_RETURN_NOT_OK_FROM_ARROW(name_builder.Append(name)); + PAIMON_RETURN_NOT_OK_FROM_ARROW(pt_builder.Append(7)); + std::shared_ptr name_array; + std::shared_ptr pt_array; + PAIMON_RETURN_NOT_OK_FROM_ARROW(name_builder.Finish(&name_array)); + PAIMON_RETURN_NOT_OK_FROM_ARROW(pt_builder.Finish(&pt_array)); + PAIMON_ASSIGN_OR_RAISE_FROM_ARROW( + std::shared_ptr struct_array, + arrow::StructArray::Make({name_array, pt_array}, int_schema->fields())); + auto c_array = std::make_unique(); + PAIMON_RETURN_NOT_OK_FROM_ARROW(arrow::ExportArray(*struct_array, c_array.get())); + RecordBatchBuilder builder(c_array.get()); + builder.SetPartition({{"pt", declared}}); + return builder.Finish(); + }; + + ASSERT_OK_AND_ASSIGN(std::unique_ptr write, + FormatTableWrite::Create(table, /*pool=*/nullptr)); + // `007` and `7` are the same partition once read into the column type, so both batches belong + // to one directory however the caller spelled the value. + ASSERT_OK_AND_ASSIGN(std::unique_ptr padded, make_batch("alice", "007")); + ASSERT_OK(write->Write(std::move(padded))); + ASSERT_OK_AND_ASSIGN(std::unique_ptr plain, make_batch("bob", "7")); + ASSERT_OK(write->Write(std::move(plain))); + ASSERT_OK_AND_ASSIGN(std::vector messages, write->PrepareCommit()); + + // One partition, so one open file and one message, named after the value rather than after + // either spelling. + ASSERT_EQ(messages.size(), 1u); + ASSERT_EQ(messages[0].partition, (std::map{{"pt", "7"}})); + ASSERT_EQ(messages[0].record_count, 2); + ASSERT_EQ(PathUtil::GetName(PathUtil::GetParentDirPath(messages[0].file_path)), "pt=7"); + + ASSERT_OK_AND_ASSIGN( + std::unique_ptr commit, + FormatTableCommit::Create(table, /*overwrite=*/false, /*static_partition=*/{})); + ASSERT_OK(commit->Commit(messages)); + ASSERT_OK_AND_ASSIGN( + std::unique_ptr scan, + FormatTableScan::Create(table, /*partition_filter=*/{}, /*limit=*/std::nullopt)); + // Aliased, or the comma inside the type would read as a second macro argument. + using PartitionList = std::vector>; + ASSERT_OK_AND_ASSIGN(PartitionList partitions, scan->ListPartitions()); + ASSERT_EQ(partitions, (PartitionList{{{"pt", "7"}}})); +} + +TEST(FormatTableTest, TestLegacyPartitionNameDecidesHowADateIsWritten) { + // `partition.legacy-name` decides how the declared partition is rendered back out, and DATE + // is the only partition type allowed here that reads back differently under it: the day count + // on, `YYYY-MM-DD` off. The table decides the directory, not the caller's spelling. + constexpr int32_t kDaysTo20240101 = 19723; + std::unique_ptr dir = UniqueTestDirectory::Create(); + ASSERT_TRUE(dir); + auto date_schema = + arrow::schema({arrow::field("id", arrow::int32()), arrow::field("dt", arrow::date32())}); + + // `kDaysTo20240101` is a constant expression, so it is read without being captured. + auto write_one = [&dir, &date_schema]( + const std::string& path, + const std::map& extra_options, + const std::string& declared) -> Result { + std::map options = {{Options::TYPE, "format-table"}, + {Options::FILE_FORMAT, "parquet"}}; + for (const auto& [key, value] : extra_options) { + options[key] = value; + } + SchemaManager schema_manager(dir->GetFileSystem(), path); + PAIMON_ASSIGN_OR_RAISE([[maybe_unused]] std::unique_ptr table_schema, + schema_manager.CreateTable(date_schema, /*partition_keys=*/{"dt"}, + /*primary_keys=*/{}, options)); + PAIMON_ASSIGN_OR_RAISE( + std::shared_ptr table, + FormatTable::Create(dir->GetFileSystem(), path, Identifier("db", "tbl"))); + + arrow::Int32Builder id_builder; + arrow::Date32Builder dt_builder; + PAIMON_RETURN_NOT_OK_FROM_ARROW(id_builder.Append(1)); + PAIMON_RETURN_NOT_OK_FROM_ARROW(dt_builder.Append(kDaysTo20240101)); + std::shared_ptr id_array; + std::shared_ptr dt_array; + PAIMON_RETURN_NOT_OK_FROM_ARROW(id_builder.Finish(&id_array)); + PAIMON_RETURN_NOT_OK_FROM_ARROW(dt_builder.Finish(&dt_array)); + PAIMON_ASSIGN_OR_RAISE_FROM_ARROW( + std::shared_ptr struct_array, + arrow::StructArray::Make({id_array, dt_array}, date_schema->fields())); + auto c_array = std::make_unique(); + PAIMON_RETURN_NOT_OK_FROM_ARROW(arrow::ExportArray(*struct_array, c_array.get())); + RecordBatchBuilder batch_builder(c_array.get()); + batch_builder.SetPartition({{"dt", declared}}); + PAIMON_ASSIGN_OR_RAISE(std::unique_ptr batch, batch_builder.Finish()); + + PAIMON_ASSIGN_OR_RAISE(std::unique_ptr write, + FormatTableWrite::Create(table, /*pool=*/nullptr)); + PAIMON_RETURN_NOT_OK(write->Write(std::move(batch))); + PAIMON_ASSIGN_OR_RAISE(std::vector messages, write->PrepareCommit()); + if (messages.size() != 1) { + return Status::Invalid("expected exactly one written file"); + } + FormatCommitMessage message = messages[0]; + PAIMON_RETURN_NOT_OK(write->Abort()); + return message; + }; + + // On by default, as everywhere else in paimon: a DATE reads back as its day count, so both + // spellings name `dt=19723` and the commit message says so too. + for (const char* declared : {"19723", "2024-01-01"}) { + ASSERT_OK_AND_ASSIGN(FormatCommitMessage message, + write_one(dir->Str() + "/legacy-" + declared, {}, declared)); + ASSERT_EQ(message.partition, (std::map{{"dt", "19723"}})) + << declared; + ASSERT_EQ(PathUtil::GetName(PathUtil::GetParentDirPath(message.file_path)), "dt=19723") + << declared; + } + + // Off, and the same two spellings name `dt=2024-01-01` instead. + const std::map not_legacy = { + {Options::PARTITION_GENERATE_LEGACY_NAME, "false"}}; + for (const char* declared : {"19723", "2024-01-01"}) { + ASSERT_OK_AND_ASSIGN(FormatCommitMessage message, + write_one(dir->Str() + "/iso-" + declared, not_legacy, declared)); + ASSERT_EQ(message.partition, (std::map{{"dt", "2024-01-01"}})) + << declared; + ASSERT_EQ(PathUtil::GetName(PathUtil::GetParentDirPath(message.file_path)), "dt=2024-01-01") + << declared; + } + + // A value that is not a date at all still has nothing to name a directory after. + ASSERT_NOK(write_one(dir->Str() + "/nonsense", {}, "not-a-date")); +} + +TEST(FormatTableTest, TestWriteRejectsWrongPartitionSpec) { + std::unique_ptr dir = UniqueTestDirectory::Create(); + ASSERT_TRUE(dir); + ASSERT_OK_AND_ASSIGN(std::shared_ptr table, + CreateTable(dir->GetFileSystem(), dir->Str(), {"dt"})); + // The batch names no partition, but the table is partitioned by one field. + ASSERT_OK_AND_ASSIGN(std::unique_ptr batch, + MakeBatch({1}, {"alice"}, "20240101", {})); + ASSERT_OK_AND_ASSIGN(std::unique_ptr write, + FormatTableWrite::Create(table, /*pool=*/nullptr)); + Status status = write->Write(std::move(batch)); + ASSERT_FALSE(status.ok()); + ASSERT_TRUE(status.IsInvalid()); + ASSERT_OK(write->Abort()); +} + +} // namespace paimon::test diff --git a/src/paimon/core/table/format/format_table_write.cpp b/src/paimon/core/table/format/format_table_write.cpp new file mode 100644 index 00000000..c62de480 --- /dev/null +++ b/src/paimon/core/table/format/format_table_write.cpp @@ -0,0 +1,656 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include "paimon/core/table/format/format_table_write.h" + +#include +#include +#include +#include +#include +#include +#include + +#include "arrow/api.h" +#include "arrow/c/bridge.h" +#include "arrow/c/helpers.h" +#include "fmt/format.h" +#include "paimon/common/data/binary_row.h" +#include "paimon/common/utils/arrow/arrow_utils.h" +#include "paimon/common/utils/arrow/mem_utils.h" +#include "paimon/common/utils/arrow/status_utils.h" +#include "paimon/common/utils/binary_row_partition_computer.h" +#include "paimon/common/utils/checked_cast.h" +#include "paimon/common/utils/field_type_utils.h" +#include "paimon/common/utils/hadoop_compression.h" +#include "paimon/common/utils/path_util.h" +#include "paimon/common/utils/scope_guard.h" +#include "paimon/common/utils/string_utils.h" +#include "paimon/core/casting/cast_executor.h" +#include "paimon/core/casting/cast_executor_factory.h" +#include "paimon/core/casting/casting_utils.h" +#include "paimon/core/core_options.h" +#include "paimon/core/table/format/format_file_naming.h" +#include "paimon/core/table/format/format_path_validation.h" +#include "paimon/defs.h" +#include "paimon/format/file_format.h" +#include "paimon/format/file_format_factory.h" +#include "paimon/format/format_writer.h" +#include "paimon/format/writer_builder.h" +#include "paimon/fs/file_system.h" +#include "paimon/logging.h" + +namespace paimon { + +namespace { + +Logger* WriteLogger() { + static std::unique_ptr logger = Logger::GetLogger("FormatTableWrite"); + return logger.get(); +} + +/// The extension a compression adds to a data file's name: a hadoop compression by its own +/// extension, anything else by the option's text. +std::string CompressionFileExtension(const std::string& compression) { + if (compression.empty()) { + return std::string(); + } + std::optional kind = HadoopCompression::FromName(compression); + if (kind) { + return HadoopCompression::ToFileExtension(*kind); + } + return compression; +} + +/// Renders a partition column as the text a partition directory is named with. +Result> RenderPartitionColumnAsText( + const std::shared_ptr& column, const std::string& field_name, + bool legacy_partition_name, arrow::MemoryPool* pool) { + if (column->type_id() == arrow::Type::STRING) { + return checked_pointer_cast(column); + } + std::shared_ptr source = column; + // `partition.legacy-name` renders with the type's own `toString`, which for a DATE is the + // day count rather than `YYYY-MM-DD`. DATE is the only partition type the two disagree on. + // `DataConverterUtils` answers it the same way for the managed table path. + if (legacy_partition_name && column->type_id() == arrow::Type::DATE32) { + PAIMON_ASSIGN_OR_RAISE( + source, + CastingUtils::Cast(column, arrow::int32(), arrow::compute::CastOptions::Safe(), pool)); + } + PAIMON_ASSIGN_OR_RAISE(FieldType source_type, + FieldTypeUtils::ConvertToFieldType(source->type()->id())); + std::shared_ptr cast_executor = + CastExecutorFactory::GetCastExecutorFactory()->GetCastExecutor(source_type, + FieldType::STRING); + if (cast_executor == nullptr) { + return Status::NotImplemented( + fmt::format("cannot name a partition directory after field '{}' of type {}", field_name, + column->type()->ToString())); + } + PAIMON_ASSIGN_OR_RAISE(std::shared_ptr casted, + cast_executor->Cast(source, arrow::utf8(), pool)); + return checked_pointer_cast(casted); +} + +} // namespace + +/// Where one partition's files go, and the partition values that directory spells out. +struct FormatTablePartitionTarget { + std::string directory; + /// The partition as the table renders it, which need not be how the caller spelled it. The + /// commit message carries these, so it cannot disagree with the directory its file sits in. + std::map partition; +}; + +/// The file currently being written for one partition. +struct FormatTableWriteFile { + std::shared_ptr out; + std::unique_ptr writer; + std::string temp_file_path; + std::string file_path; + int64_t record_count = 0; +}; + +class FormatTableWrite::Impl { + public: + /// Where a partition's files belong. Cached, since deriving it reads the values into their + /// column types, renders them back out, escapes them and re-walks a whole path. + Result GetPartitionTarget( + const std::map& partition); + + /// Opens a new hidden file in `directory` for the partition that directory stands for. + Result OpenFile(const std::string& directory); + + /// Closes the file open in `directory` and records it for committing. A failure leaves no + /// open file behind and stops the write, since the file is then neither writable nor + /// publishable. + Status FinishFile(const std::string& directory); + + /// Closes `file` and appends the message that publishes it. Everything that can fail lives + /// here, so `FinishFile()` has one place to clean up after. + Status CloseFileAndStage(FormatTableWriteFile* file, + const std::map& partition); + + /// Gives up on a file that was never staged: closes what is still open and removes the temp + /// file. Best effort, like `Abort()`, since the failure that got here is the one to report. + void DiscardOpenFile(FormatTableWriteFile* file); + + /// Checks that every row belongs to the partition the batch declares. Partition columns are + /// not written, so a disagreeing row would read back with the declared value and lose its + /// own. + Status ValidatePartitionColumns( + const std::shared_ptr& batch, + const std::vector>& ordered_partition); + + std::shared_ptr table; + std::shared_ptr pool; + std::unique_ptr arrow_pool; + /// Full table schema, used to check the incoming batch. + std::shared_ptr table_schema; + std::shared_ptr table_struct_type; + /// Columns actually stored in the files: the table's, minus the partition ones. + std::shared_ptr data_schema; + /// Index in the table schema of each data column. + std::vector data_column_indexes; + /// Index in the table schema of each partition column, in partition key order. + std::vector partition_column_indexes; + std::string format_identifier; + std::string file_compression; + /// From `partition.legacy-name`. It decides how a row's partition column is rendered, so the + /// directory name and the row check both go by it. + bool legacy_partition_name = true; + /// Reads a partition into its column types and renders it back out, the way Java Paimon's + /// writer does. Null when the table is not partitioned. + std::unique_ptr partition_computer; + int64_t target_file_size = 0; + int64_t target_file_row_num = 0; + int32_t write_batch_size = 0; + FormatFileNaming naming; + + /// Keyed by the partition the caller declared. See `GetPartitionTarget()`. + std::map, FormatTablePartitionTarget> partition_targets; + /// Open file per partition, keyed by the partition's directory. + std::map open_files; + /// Partition values of each open file, by the same key. + std::map> open_partitions; + /// Written and closed but not yet published. Kept after `PrepareCommit()` hands out a copy, + /// so that an `Abort()` still knows what to remove. + std::vector staged_messages; + bool prepared = false; + bool aborted = false; + /// The failure that closing a file stopped at, or OK. It ends the write: the rows of that + /// file cannot be published, so publishing the others would quietly lose them. + Status finish_failure; + + /// Why this write will take no more rows, or null while it still will. Prepared and aborted + /// call for different work from the caller, so the refusal names which one it is. + const char* FinishedReason() const { + if (aborted) { + return "format table write has been aborted"; + } + if (prepared) { + return "format table write has already prepared its commit"; + } + return nullptr; + } +}; + +FormatTableWrite::FormatTableWrite(std::unique_ptr impl) : impl_(std::move(impl)) {} + +FormatTableWrite::~FormatTableWrite() { + if (impl_ != nullptr && impl_->FinishedReason() == nullptr) { + // `Abort()` logs its cleanup failures and returns OK today; the status is still checked. + Status status = Abort(); + if (!status.ok()) { + PAIMON_LOG_WARN(WriteLogger(), "Failed to abort an abandoned write of table %s: %s", + impl_->table->FullName().c_str(), status.ToString().c_str()); + } + } +} + +Result> FormatTableWrite::Create( + const std::shared_ptr& table, const std::shared_ptr& pool) { + if (table == nullptr) { + return Status::Invalid("format table write requires a table"); + } + auto impl = std::make_unique(); + impl->table = table; + impl->pool = pool != nullptr ? pool : GetDefaultPool(); + + PAIMON_ASSIGN_OR_RAISE(std::unique_ptr<::ArrowSchema> c_schema, table->GetArrowSchema()); + PAIMON_ASSIGN_OR_RAISE_FROM_ARROW(impl->table_schema, arrow::ImportSchema(c_schema.get())); + impl->table_struct_type = arrow::struct_(impl->table_schema->fields()); + + const std::vector& partition_keys = table->PartitionKeys(); + arrow::FieldVector data_fields; + for (int32_t i = 0; i < impl->table_schema->num_fields(); i++) { + const std::shared_ptr& field = impl->table_schema->field(i); + if (std::find(partition_keys.begin(), partition_keys.end(), field->name()) == + partition_keys.end()) { + data_fields.push_back(field); + impl->data_column_indexes.push_back(i); + } + } + if (data_fields.empty()) { + return Status::Invalid(fmt::format( + "format table {} has no non-partition column, so its files would hold nothing", + table->FullName())); + } + impl->data_schema = arrow::schema(data_fields); + + // In partition key order, which is the order the directories nest in. + for (const std::string& partition_key : partition_keys) { + int32_t index = impl->table_schema->GetFieldIndex(partition_key); + if (index < 0) { + return Status::Invalid(fmt::format("partition field '{}' is not a column of table {}", + partition_key, table->FullName())); + } + impl->partition_column_indexes.push_back(index); + } + + impl->format_identifier = FormatTable::FormatToString(table->GetFormat()); + impl->file_compression = table->FileCompression(); + + // Through `CoreOptions`, so `"256 mb"` means what it does elsewhere and a default lives in + // one place. + PAIMON_ASSIGN_OR_RAISE(CoreOptions core_options, + CoreOptions::FromMap(table->Options(), table->GetFileSystem())); + + // parquet and orc record their compression inside the file, so the name keeps the plain + // `.parquet` unless `file.suffix.include.compression` asks for it, and then the compression + // goes in front: `data--0.snappy.parquet`. + const std::string compression_extension = CompressionFileExtension(impl->file_compression); + std::string extension = impl->format_identifier; + if (!compression_extension.empty() && core_options.FileSuffixIncludeCompression()) { + extension = compression_extension + "." + extension; + } + + impl->legacy_partition_name = core_options.LegacyPartitionNameEnabled(); + if (!partition_keys.empty()) { + PAIMON_ASSIGN_OR_RAISE( + impl->partition_computer, + BinaryRowPartitionComputer::Create(partition_keys, impl->table_schema, + table->PartitionDefaultName(), + impl->legacy_partition_name, impl->pool)); + } + // A format table has no primary keys, so its target file size is the append-table default. + impl->target_file_size = core_options.GetTargetFileSize(/*has_primary_key=*/false); + impl->target_file_row_num = core_options.GetTargetFileRowNum(); + impl->write_batch_size = core_options.GetWriteBatchSize(); + impl->arrow_pool = GetArrowPool(impl->pool); + PAIMON_ASSIGN_OR_RAISE(impl->naming, + FormatFileNaming::Create(extension, core_options.DataFilePrefix())); + + return std::unique_ptr(new FormatTableWrite(std::move(impl))); +} + +Result FormatTableWrite::Impl::GetPartitionTarget( + const std::map& partition) { + auto iter = partition_targets.find(partition); + if (iter != partition_targets.end()) { + return iter->second; + } + FormatTablePartitionTarget target; + target.partition = partition; + if (partition_computer != nullptr) { + // The round trip Java Paimon's writer makes when it renders a partition out of a row: the + // values are read into their column types and rendered back, so the directory is named the + // way the table's options say rather than the way the caller spelled the value. + PAIMON_ASSIGN_OR_RAISE(BinaryRow row, partition_computer->ToBinaryRow(partition)); + // Aliased, or the comma inside the type would read as a second macro argument. + using RenderedPartition = std::vector>; + PAIMON_ASSIGN_OR_RAISE(RenderedPartition rendered, + partition_computer->GeneratePartitionVector(row)); + target.partition.clear(); + for (auto& [key, value] : rendered) { + target.partition.emplace(std::move(key), std::move(value)); + } + } + PAIMON_ASSIGN_OR_RAISE(target.directory, + FormatPathValidation::BuildPartitionDirectory(table, target.partition)); + partition_targets.emplace(partition, target); + return target; +} + +Status FormatTableWrite::Impl::ValidatePartitionColumns( + const std::shared_ptr& batch, + const std::vector>& ordered_partition) { + // A null, an empty string and a whitespace-only string alike stand for the default partition + // name, so all three land in the same directory. + const std::string& default_partition_name = table->PartitionDefaultName(); + for (size_t i = 0; i < ordered_partition.size(); i++) { + const std::string& partition_key = ordered_partition[i].first; + const std::string& declared_value = ordered_partition[i].second; + PAIMON_ASSIGN_OR_RAISE( + std::shared_ptr text_column, + RenderPartitionColumnAsText(batch->field(partition_column_indexes[i]), partition_key, + legacy_partition_name, arrow_pool.get())); + // False for every partition the table rendered, since `GeneratePartitionVector()` has + // already replaced a null or blank value with the default partition name. Kept so that a + // value reaching here without that round trip cannot take the fast path below, where a + // blank has to compare equal to the default partition name rather than to itself. + const bool declared_is_blank = StringUtils::IsNullOrWhitespaceOnly(declared_value); + for (int64_t row = 0; row < text_column->length(); row++) { + const bool is_null = text_column->IsNull(row); + const std::string_view rendered = + is_null ? std::string_view() : text_column->GetView(row); + // The ordinary case: a row that renders exactly as the batch declared. + if (!is_null && !declared_is_blank && rendered == declared_value) { + continue; + } + const std::string_view row_value = StringUtils::IsNullOrWhitespaceOnly(rendered) + ? std::string_view(default_partition_name) + : rendered; + if (row_value == declared_value) { + continue; + } + return Status::Invalid(fmt::format( + "row {} of the batch has '{}' in partition column '{}', but the batch declares " + "partition '{}={}'. The partition columns are not written to the file, so this " + "row would be stored under a partition it does not belong to and read back " + "with the declared value.", + row, row_value, partition_key, partition_key, declared_value)); + } + } + return Status::OK(); +} + +Result FormatTableWrite::Impl::OpenFile(const std::string& directory) { + FormatTableWriteFile file; + file.file_path = PathUtil::JoinPath(directory, naming.NextFileName()); + PAIMON_ASSIGN_OR_RAISE(std::string temp_relative_path, naming.NextTempFilePath()); + file.temp_file_path = PathUtil::JoinPath(directory, temp_relative_path); + + PAIMON_ASSIGN_OR_RAISE(std::unique_ptr file_format, + FileFormatFactory::Get(format_identifier, table->Options())); + ::ArrowSchema c_schema; + ArrowSchemaMarkReleased(&c_schema); + ScopeGuard schema_guard([&c_schema]() { ArrowSchemaRelease(&c_schema); }); + PAIMON_RETURN_NOT_OK_FROM_ARROW(arrow::ExportSchema(*data_schema, &c_schema)); + PAIMON_ASSIGN_OR_RAISE(std::unique_ptr writer_builder, + file_format->CreateWriterBuilder(&c_schema, write_batch_size)); + writer_builder->WithMemoryPool(pool); + + PAIMON_ASSIGN_OR_RAISE( + std::unique_ptr out, + table->GetFileSystem()->Create(file.temp_file_path, /*overwrite=*/false)); + file.out = std::move(out); + // From here the temp file exists on disk and only this guard knows about it, so a failure + // below has to remove it or it is left behind for good. + std::shared_ptr file_system = table->GetFileSystem(); + ScopeGuard temp_file_guard([&file, &file_system]() { + // The stream is closed before the file goes: a stream dropped without `Close()` may still + // flush, and on an object store that write would land after the delete. + if (file.out != nullptr) { + Status closed = file.out->Close(); + if (!closed.ok()) { + PAIMON_LOG_WARN(WriteLogger(), + "Failed to close %s after its writer could not be opened: %s", + file.temp_file_path.c_str(), closed.ToString().c_str()); + } + file.out.reset(); + } + Status status = file_system->Delete(file.temp_file_path, /*recursive=*/false); + if (!status.ok()) { + PAIMON_LOG_WARN(WriteLogger(), + "Failed to remove the temp file %s after its writer " + "could not be opened: %s", + file.temp_file_path.c_str(), status.ToString().c_str()); + } + }); + PAIMON_ASSIGN_OR_RAISE(file.writer, writer_builder->Build(file.out, file_compression)); + temp_file_guard.Release(); + return file; +} + +Status FormatTableWrite::Impl::FinishFile(const std::string& directory) { + auto file_iter = open_files.find(directory); + if (file_iter == open_files.end()) { + return Status::OK(); + } + // The file stays in `open_files` until it is recorded for committing: dropped earlier, a + // failure below would leave a temp path no `Abort()` knows about. + FormatTableWriteFile& file = file_iter->second; + // The partition is written and erased together with the file, so this cannot miss; should + // that ever break, it must not quietly commit a file under no partition. + auto partition_iter = open_partitions.find(directory); + if (partition_iter == open_partitions.end()) { + return Status::Invalid( + fmt::format("no partition was recorded for the file open in {}", directory)); + } + + Status status = CloseFileAndStage(&file, partition_iter->second); + if (!status.ok()) { + // Closing gets this far only once the writer is finished and gone, so there is nothing + // left to write into and nothing whole to publish. What is on disk is discarded and the + // write stops here: leaving the entry in `open_files` would have the next `Write()` or + // `PrepareCommit()` reach through a writer that is no longer there. + DiscardOpenFile(&file); + open_files.erase(file_iter); + open_partitions.erase(partition_iter); + finish_failure = status; + return status; + } + open_files.erase(file_iter); + open_partitions.erase(partition_iter); + return Status::OK(); +} + +Status FormatTableWrite::Impl::CloseFileAndStage( + FormatTableWriteFile* file, const std::map& partition) { + PAIMON_RETURN_NOT_OK(file->writer->Flush()); + PAIMON_RETURN_NOT_OK(file->writer->Finish()); + file->writer.reset(); + // The size is read before closing, while the stream still knows how far it wrote. + PAIMON_ASSIGN_OR_RAISE(int64_t file_size, file->out->GetPos()); + PAIMON_RETURN_NOT_OK(file->out->Flush()); + PAIMON_RETURN_NOT_OK(file->out->Close()); + file->out.reset(); + + PAIMON_LOG_DEBUG(WriteLogger(), "Staged %s for %s, %ld rows, %ld bytes", + file->temp_file_path.c_str(), file->file_path.c_str(), file->record_count, + file_size); + staged_messages.emplace_back(file->temp_file_path, file->file_path, partition, + file->record_count, file_size); + return Status::OK(); +} + +void FormatTableWrite::Impl::DiscardOpenFile(FormatTableWriteFile* file) { + std::shared_ptr file_system = table->GetFileSystem(); + if (file->writer != nullptr) { + Status status = file->writer->Finish(); + if (!status.ok()) { + PAIMON_LOG_WARN(WriteLogger(), "Failed to finish the writer of %s while discarding: %s", + file->temp_file_path.c_str(), status.ToString().c_str()); + } + file->writer.reset(); + } + // The stream is closed before the file goes: a stream dropped without `Close()` may still + // flush, and on an object store that write would land after the delete. + if (file->out != nullptr) { + Status status = file->out->Close(); + if (!status.ok()) { + PAIMON_LOG_WARN(WriteLogger(), "Failed to close %s while discarding: %s", + file->temp_file_path.c_str(), status.ToString().c_str()); + } + file->out.reset(); + } + Status status = file_system->Delete(file->temp_file_path, /*recursive=*/false); + if (!status.ok() && !status.IsNotExist()) { + PAIMON_LOG_WARN(WriteLogger(), "Failed to remove the temp file %s while discarding: %s", + file->temp_file_path.c_str(), status.ToString().c_str()); + } +} + +Status FormatTableWrite::Write(std::unique_ptr&& batch) { + if (const char* finished = impl_->FinishedReason(); finished != nullptr) { + return Status::Invalid(finished); + } + // A file that could not be closed ends the write: see `Impl::finish_failure`. + PAIMON_RETURN_NOT_OK(impl_->finish_failure); + if (batch == nullptr || batch->GetData() == nullptr) { + return Status::Invalid("format table write requires a batch"); + } + for (RecordBatch::RowKind row_kind : batch->GetRowKind()) { + if (row_kind != RecordBatch::RowKind::INSERT) { + return Status::Invalid( + "format table only supports INSERT rows: a directory of data files records no " + "row identity for an update or a delete to apply to"); + } + } + + // A partial partition would not name a single directory. + const std::map& partition = batch->GetPartition(); + const std::vector& partition_keys = impl_->table->PartitionKeys(); + if (partition.size() != partition_keys.size()) { + return Status::Invalid(fmt::format( + "batch carries {} partition values but table {} is partitioned by {} fields", + partition.size(), impl_->table->FullName(), partition_keys.size())); + } + for (const std::string& partition_key : partition_keys) { + if (partition.find(partition_key) == partition.end()) { + return Status::Invalid(fmt::format( + "batch does not carry a value for partition field '{}'", partition_key)); + } + } + + // `year=2025/month=01/`, or `2025/01/` under `format-table.partition-path-only-value`. The + // scan reads back whichever is written here, so both go through one place. + PAIMON_ASSIGN_OR_RAISE(FormatTablePartitionTarget target, impl_->GetPartitionTarget(partition)); + const std::string& directory = target.directory; + // In the partition the table rendered rather than the one the caller spelled, since a value + // can be written more than one way. + std::vector> ordered_partition; + ordered_partition.reserve(partition_keys.size()); + for (const std::string& partition_key : partition_keys) { + auto iter = target.partition.find(partition_key); + if (iter == target.partition.end()) { + return Status::Invalid( + fmt::format("partition field '{}' is missing from the partition the table rendered", + partition_key)); + } + ordered_partition.emplace_back(partition_key, iter->second); + } + + PAIMON_ASSIGN_OR_RAISE_FROM_ARROW( + std::shared_ptr array, + arrow::ImportArray(batch->GetData(), impl_->table_struct_type)); + PAIMON_RETURN_NOT_OK(ArrowUtils::CheckNullabilityMatch(impl_->table_schema, array)); + auto table_struct = checked_pointer_cast(array); + if (table_struct->null_count() != 0) { + return Status::Invalid( + "format table write does not support a null row: a row of the table must have a value " + "for every column, even if that value is null"); + } + + PAIMON_RETURN_NOT_OK(impl_->ValidatePartitionColumns(table_struct, ordered_partition)); + + // Partition columns are not written: the directory holds those values. + arrow::ArrayVector data_columns; + data_columns.reserve(impl_->data_column_indexes.size()); + for (int32_t index : impl_->data_column_indexes) { + data_columns.push_back(table_struct->field(index)); + } + PAIMON_ASSIGN_OR_RAISE_FROM_ARROW( + std::shared_ptr data_struct, + arrow::StructArray::Make(data_columns, impl_->data_schema->fields())); + + // A write that never receives a row leaves the directory as it found it. + if (data_struct->length() == 0) { + return Status::OK(); + } + + auto file_iter = impl_->open_files.find(directory); + if (file_iter == impl_->open_files.end()) { + PAIMON_ASSIGN_OR_RAISE(FormatTableWriteFile file, impl_->OpenFile(directory)); + file_iter = impl_->open_files.emplace(directory, std::move(file)).first; + impl_->open_partitions[directory] = target.partition; + } + + ArrowArray c_data_array; + PAIMON_RETURN_NOT_OK_FROM_ARROW(arrow::ExportArray(*data_struct, &c_data_array)); + PAIMON_RETURN_NOT_OK(file_iter->second.writer->AddBatch(&c_data_array)); + file_iter->second.record_count += data_struct->length(); + + // Checked between batches, so a file may pass either target by up to one batch. The row count + // comes first, since it costs the writer nothing. + bool reached = file_iter->second.record_count >= impl_->target_file_row_num; + if (!reached) { + PAIMON_ASSIGN_OR_RAISE(reached, file_iter->second.writer->ReachTargetSize( + /*suggested_check=*/true, impl_->target_file_size)); + } + if (reached) { + PAIMON_RETURN_NOT_OK(impl_->FinishFile(directory)); + } + return Status::OK(); +} + +Result> FormatTableWrite::PrepareCommit() { + if (const char* finished = impl_->FinishedReason(); finished != nullptr) { + return Status::Invalid(finished); + } + // A file that could not be closed ends the write: see `Impl::finish_failure`. + PAIMON_RETURN_NOT_OK(impl_->finish_failure); + std::vector directories; + directories.reserve(impl_->open_files.size()); + for (const auto& open_file : impl_->open_files) { + directories.push_back(open_file.first); + } + for (const std::string& directory : directories) { + Status status = impl_->FinishFile(directory); + if (!status.ok()) { + // Whatever was already closed is unpublished too, so nothing of this write survives. + Status abort_status = Abort(); + if (!abort_status.ok()) { + PAIMON_LOG_WARN(WriteLogger(), + "Failed to abort table %s after preparing its commit failed: %s", + impl_->table->FullName().c_str(), abort_status.ToString().c_str()); + } + return status; + } + } + impl_->prepared = true; + // A copy, not a move: `Abort()` must still know what to remove afterwards. + return impl_->staged_messages; +} + +Status FormatTableWrite::Abort() { + std::shared_ptr file_system = impl_->table->GetFileSystem(); + // Best effort: a file that cannot be removed stays behind, hidden, rather than failing the + // abort and hiding whatever caused it. + for (auto& open_file : impl_->open_files) { + impl_->DiscardOpenFile(&open_file.second); + } + impl_->open_files.clear(); + impl_->open_partitions.clear(); + for (const FormatCommitMessage& message : impl_->staged_messages) { + Status status = file_system->Delete(message.temp_file_path, /*recursive=*/false); + // Already gone is the ordinary case once the files have been committed, or aborted once. + if (!status.ok() && !status.IsNotExist()) { + PAIMON_LOG_WARN(WriteLogger(), "Failed to remove the staged file %s while aborting: %s", + message.temp_file_path.c_str(), status.ToString().c_str()); + } + } + impl_->staged_messages.clear(); + // Kept apart from `prepared`, so a later call reports aborted rather than committed. + impl_->aborted = true; + return Status::OK(); +} + +} // namespace paimon diff --git a/src/paimon/core/table/format/format_table_write.h b/src/paimon/core/table/format/format_table_write.h new file mode 100644 index 00000000..7d03d696 --- /dev/null +++ b/src/paimon/core/table/format/format_table_write.h @@ -0,0 +1,85 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#pragma once + +#include +#include + +#include "paimon/core/table/format/format_commit_message.h" +#include "paimon/memory/memory_pool.h" +#include "paimon/record_batch.h" +#include "paimon/result.h" +#include "paimon/status.h" +#include "paimon/table/format/format_table.h" + +namespace paimon { + +/// Writes new rows into a format table. +/// +/// Only inserts are supported: a directory of plain data files has nowhere to record that a row +/// replaced or removed an earlier one, so a batch carrying any other row kind is rejected. +/// +/// Writing is two-phase, because a directory has no metadata to switch atomically: `Write()` fills +/// files in a `_temporary` directory beside where they will end up, which a scan skips, and only +/// `FormatTableCommit` renames them into place. That is the layout Java Paimon stages under too. +/// +/// A write dropped before `PrepareCommit()` clears what it staged from its destructor. One that +/// has prepared has handed those files to a commit, so its destructor leaves them alone; if that +/// commit never happens, `Abort()` removes them. +/// +/// The partition a batch belongs to comes from `RecordBatch::GetPartition()`, and the partition +/// columns are not written into the file: a Hive-style layout keeps those values in the directory +/// names. One batch therefore carries one partition, and every row is checked against it. +/// +/// Not thread-safe: one write holds an open file per partition and a counter naming them, so a +/// single thread must drive it. Separate writes may fill one table at once, since each stages its +/// files under a uuid of its own. +class FormatTableWrite { + public: + /// @param table Table to write to. + /// @param pool Memory pool the writers allocate from. + static Result> Create( + const std::shared_ptr& table, const std::shared_ptr& pool); + + ~FormatTableWrite(); + + /// Writes a batch of rows. The batch's data must match the table schema, including its + /// partition columns, whose values must also be given by `RecordBatch::SetPartition()`. + Status Write(std::unique_ptr&& batch); + + /// Closes the written files and reports them for committing. The write cannot be used + /// afterwards. + Result> PrepareCommit(); + + /// Removes every file this write has staged, on a best-effort basis. + /// + /// This is the one call still allowed after `PrepareCommit()`, and after itself, so a commit + /// that is prepared and then abandoned can still be cleaned up. A file already published by + /// `FormatTableCommit` is no longer staged and is not touched. + Status Abort(); + + class Impl; + + private: + explicit FormatTableWrite(std::unique_ptr impl); + + std::unique_ptr impl_; +}; + +} // namespace paimon diff --git a/src/paimon/core/table/format/lazy_concat_batch_reader.cpp b/src/paimon/core/table/format/lazy_concat_batch_reader.cpp new file mode 100644 index 00000000..0f807df6 --- /dev/null +++ b/src/paimon/core/table/format/lazy_concat_batch_reader.cpp @@ -0,0 +1,130 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include "paimon/core/table/format/lazy_concat_batch_reader.h" + +#include +#include + +// `ReadBatch` holds `unique_ptr`s to these, so destroying one needs their definitions. +#include "arrow/c/abi.h" +#include "fmt/format.h" +#include "paimon/common/metrics/metrics_impl.h" +#include "paimon/common/reader/reader_utils.h" +#include "paimon/common/utils/arrow/mem_utils.h" + +namespace paimon { + +namespace { +/// Says which file a failure came from, keeping the status code it carried. +Status WithFileName(const std::string& name, const Status& status) { + return Status(status.code(), fmt::format("cannot read {}: {}", name, status.message())); +} +} // namespace + +LazyConcatBatchReader::LazyConcatBatchReader(std::vector&& sources, + const std::shared_ptr& pool) + : arrow_pool_(GetArrowPool(pool)), + sources_(std::move(sources)), + closed_metrics_(std::make_shared()) {} + +LazyConcatBatchReader::~LazyConcatBatchReader() { + DoClose(); +} + +void LazyConcatBatchReader::CloseCurrent() { + if (current_reader_ == nullptr) { + return; + } + // Taken before the reader goes, or everything read through it is missing from the totals. + std::shared_ptr metrics = current_reader_->GetReaderMetrics(); + current_reader_->Close(); + if (metrics != nullptr) { + closed_metrics_->Merge(metrics); + } + // Kept until this reader goes: a batch it handed out is allocated from a pool it owns. + // Closing already released the file, so little stays behind. + closed_readers_.push_back(std::move(current_reader_)); + current_name_.clear(); +} + +Result LazyConcatBatchReader::NextBatchWithBitmap() { + // `BatchReader` promises a failure is terminal: keep answering with it rather than moving + // on to the next file. + if (!failure_.ok()) { + return failure_; + } + while (true) { + if (current_reader_ == nullptr) { + if (next_source_ >= sources_.size()) { + return BatchReader::MakeEofBatchWithBitmap(); + } + Source& source = sources_[next_source_]; + Result> opened = source.open(); + if (!opened.ok()) { + failure_ = WithFileName(source.name, opened.status()); + return failure_; + } + next_source_++; + current_reader_ = std::move(opened).value(); + if (current_reader_ == nullptr) { + continue; + } + current_name_ = source.name; + } + Result result = current_reader_->NextBatchWithBitmap(); + if (!result.ok()) { + failure_ = WithFileName(current_name_, result.status()); + return failure_; + } + if (!BatchReader::IsEofBatch(result.value())) { + return std::move(result).value(); + } + CloseCurrent(); + } +} + +Result LazyConcatBatchReader::NextBatch() { + PAIMON_ASSIGN_OR_RAISE(BatchReader::ReadBatchWithBitmap batch_with_bitmap, + NextBatchWithBitmap()); + return ReaderUtils::ApplyBitmapToReadBatch(std::move(batch_with_bitmap), arrow_pool_.get()); +} + +void LazyConcatBatchReader::Close() { + DoClose(); +} + +void LazyConcatBatchReader::DoClose() { + CloseCurrent(); + // Whatever was never reached was never opened. + next_source_ = sources_.size(); +} + +std::shared_ptr LazyConcatBatchReader::GetReaderMetrics() const { + auto metrics = std::make_shared(); + metrics->Merge(closed_metrics_); + if (current_reader_ != nullptr) { + std::shared_ptr current = current_reader_->GetReaderMetrics(); + if (current != nullptr) { + metrics->Merge(current); + } + } + return metrics; +} + +} // namespace paimon diff --git a/src/paimon/core/table/format/lazy_concat_batch_reader.h b/src/paimon/core/table/format/lazy_concat_batch_reader.h new file mode 100644 index 00000000..d8df5202 --- /dev/null +++ b/src/paimon/core/table/format/lazy_concat_batch_reader.h @@ -0,0 +1,87 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#pragma once + +#include +#include +#include +#include +#include + +#include "arrow/api.h" +#include "paimon/metrics.h" +#include "paimon/reader/batch_reader.h" +#include "paimon/result.h" +#include "paimon/status.h" + +namespace paimon { + +class MemoryPool; +class MetricsImpl; + +/// Reads a sequence of readers one after another, opening each only when it is reached. +/// +/// A split holds a whole partition's files and a reader holds its file open, so building them all +/// up front could exhaust the file descriptors before returning a row. Here one file is open at a +/// time, closed as soon as it runs out. A closed reader is kept, and its metrics with it: a batch +/// it handed out is allocated from a pool that reader owns. +/// +/// Every failure names the file it came from, which is the only thing telling one file of a +/// partition from another, and is terminal as `BatchReader` requires. +class LazyConcatBatchReader : public BatchReader { + public: + /// Opens one reader, at most once, when the previous reader runs out. + using ReaderFactory = std::function>()>; + + /// One reader and the file it reads, named in every error that reader produces. + struct Source { + std::string name; + ReaderFactory open; + }; + + LazyConcatBatchReader(std::vector&& sources, const std::shared_ptr& pool); + + ~LazyConcatBatchReader() override; + + Result NextBatch() override; + Result NextBatchWithBitmap() override; + void Close() override; + std::shared_ptr GetReaderMetrics() const override; + + private: + /// Closes every reader; called by both `Close` and the destructor. + void DoClose(); + + /// Closes the reader in hand, keeping its metrics. + void CloseCurrent(); + + std::unique_ptr arrow_pool_; + std::vector sources_; + size_t next_source_ = 0; + std::unique_ptr current_reader_; + /// The readers already closed, held so the batches they handed out stay valid. + std::vector> closed_readers_; + /// The file `current_reader_` reads, kept so a failure part way through can name it. + std::string current_name_; + /// The failure this reader stopped at, or OK while it has not failed. + Status failure_; + std::shared_ptr closed_metrics_; +}; + +} // namespace paimon diff --git a/src/paimon/core/table/format/lazy_concat_batch_reader_test.cpp b/src/paimon/core/table/format/lazy_concat_batch_reader_test.cpp new file mode 100644 index 00000000..0f471f21 --- /dev/null +++ b/src/paimon/core/table/format/lazy_concat_batch_reader_test.cpp @@ -0,0 +1,288 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include "paimon/core/table/format/lazy_concat_batch_reader.h" + +#include +#include +#include +#include + +#include "arrow/api.h" +#include "arrow/c/bridge.h" +#include "gtest/gtest.h" +#include "paimon/common/metrics/metrics_impl.h" +#include "paimon/common/reader/reader_utils.h" +#include "paimon/common/utils/arrow/status_utils.h" +#include "paimon/memory/memory_pool.h" +#include "paimon/status.h" +#include "paimon/testing/utils/testharness.h" + +namespace paimon::test { + +namespace { + +/// A reader over a fixed number of one-row batches that records when it was closed. +class OneColumnBatchReader : public BatchReader { + public: + OneColumnBatchReader(int32_t value, int32_t batches, bool* closed_flag) + : value_(value), + remaining_(batches), + closed_flag_(closed_flag), + metrics_(std::make_shared()) {} + + Result NextBatch() override { + if (remaining_-- <= 0) { + return BatchReader::MakeEofBatch(); + } + arrow::Int32Builder builder; + PAIMON_RETURN_NOT_OK_FROM_ARROW(builder.Append(value_)); + std::shared_ptr column; + PAIMON_RETURN_NOT_OK_FROM_ARROW(builder.Finish(&column)); + PAIMON_ASSIGN_OR_RAISE_FROM_ARROW( + std::shared_ptr struct_array, + arrow::StructArray::Make({column}, std::vector{"id"})); + auto c_array = std::make_unique(); + auto c_schema = std::make_unique(); + PAIMON_RETURN_NOT_OK_FROM_ARROW( + arrow::ExportArray(*struct_array, c_array.get(), c_schema.get())); + return std::make_pair(std::move(c_array), std::move(c_schema)); + } + + std::shared_ptr GetReaderMetrics() const override { + return metrics_; + } + + void Close() override { + if (closed_flag_ != nullptr) { + *closed_flag_ = true; + } + } + + private: + int32_t value_; + int32_t remaining_; + bool* closed_flag_; + std::shared_ptr metrics_; +}; + +/// A reader that fails on its first batch, standing for a file that opens and then turns out to +/// be unreadable - a truncated record, a value no type can hold. +class FailingBatchReader : public BatchReader { + public: + explicit FailingBatchReader(Status failure) + : failure_(std::move(failure)), metrics_(std::make_shared()) {} + + Result NextBatch() override { + return failure_; + } + + std::shared_ptr GetReaderMetrics() const override { + return metrics_; + } + + void Close() override {} + + private: + Status failure_; + std::shared_ptr metrics_; +}; + +/// Names factories "file-0", "file-1", ... A source carries the name of the file it reads so a +/// failure can point at one; which name it is does not matter to these tests. +std::vector SourcesOf( + std::vector&& factories) { + std::vector sources; + sources.reserve(factories.size()); + for (size_t i = 0; i < factories.size(); i++) { + sources.push_back({"file-" + std::to_string(i), std::move(factories[i])}); + } + return sources; +} + +int64_t DrainRows(BatchReader* reader) { + int64_t rows = 0; + while (true) { + Result batch = reader->NextBatch(); + if (!batch.ok() || BatchReader::IsEofBatch(batch.value())) { + return rows; + } + rows += batch.value().first->length; + ReaderUtils::ReleaseReadBatch(std::move(batch).value()); + } +} + +} // namespace + +TEST(LazyConcatBatchReaderTest, TestAReaderIsOpenedOnlyWhenItIsReached) { + // A split holds a whole partition's files, and a reader keeps its file open. Building them all + // up front would hold one descriptor per file before a single row came back. + int32_t opened = 0; + std::vector factories; + for (int32_t i = 0; i < 3; i++) { + factories.push_back([&opened, i]() -> Result> { + opened++; + return std::make_unique(i, /*batches=*/1, nullptr); + }); + } + LazyConcatBatchReader reader(SourcesOf(std::move(factories)), GetDefaultPool()); + ASSERT_EQ(opened, 0); + + ASSERT_OK_AND_ASSIGN(BatchReader::ReadBatch first, reader.NextBatch()); + ASSERT_FALSE(BatchReader::IsEofBatch(first)); + ReaderUtils::ReleaseReadBatch(std::move(first)); + ASSERT_EQ(opened, 1); + ASSERT_EQ(DrainRows(&reader), 2); + ASSERT_EQ(opened, 3); +} + +TEST(LazyConcatBatchReaderTest, TestAReaderIsClosedAsSoonAsItRunsOut) { + bool first_closed = false; + bool second_closed = false; + std::vector factories; + factories.push_back([&first_closed]() -> Result> { + return std::make_unique(0, /*batches=*/1, &first_closed); + }); + factories.push_back([&second_closed]() -> Result> { + return std::make_unique(1, /*batches=*/1, &second_closed); + }); + LazyConcatBatchReader reader(SourcesOf(std::move(factories)), GetDefaultPool()); + + ASSERT_OK_AND_ASSIGN(BatchReader::ReadBatch first, reader.NextBatch()); + ASSERT_FALSE(BatchReader::IsEofBatch(first)); + ReaderUtils::ReleaseReadBatch(std::move(first)); + ASSERT_FALSE(first_closed); + // Reaching the second file's rows means the first file is done with, so it is let go at once + // rather than at the end of the split. + ASSERT_OK_AND_ASSIGN(BatchReader::ReadBatch second, reader.NextBatch()); + ASSERT_FALSE(BatchReader::IsEofBatch(second)); + ReaderUtils::ReleaseReadBatch(std::move(second)); + ASSERT_TRUE(first_closed); + ASSERT_FALSE(second_closed); +} + +TEST(LazyConcatBatchReaderTest, TestClosingDoesNotOpenWhatWasNeverReached) { + int32_t opened = 0; + std::vector factories; + for (int32_t i = 0; i < 3; i++) { + factories.push_back([&opened, i]() -> Result> { + opened++; + return std::make_unique(i, /*batches=*/1, nullptr); + }); + } + LazyConcatBatchReader reader(SourcesOf(std::move(factories)), GetDefaultPool()); + ASSERT_OK_AND_ASSIGN(BatchReader::ReadBatch first, reader.NextBatch()); + ASSERT_FALSE(BatchReader::IsEofBatch(first)); + ReaderUtils::ReleaseReadBatch(std::move(first)); + reader.Close(); + ASSERT_EQ(opened, 1); + // And nothing is opened afterwards either. + ASSERT_OK_AND_ASSIGN(BatchReader::ReadBatch eof, reader.NextBatch()); + ASSERT_TRUE(BatchReader::IsEofBatch(eof)); + ASSERT_EQ(opened, 1); +} + +TEST(LazyConcatBatchReaderTest, TestAFailureIsTerminal) { + // `BatchReader` says a failure must not be retried: the reader keeps answering with the same + // error. Moving on to the next file would let a caller that ignored the error read a + // partition with a file silently missing from it. + int32_t opened = 0; + std::vector factories; + factories.push_back([&opened]() -> Result> { + opened++; + return Status::IOError("cannot open the file"); + }); + factories.push_back([&opened]() -> Result> { + opened++; + return std::make_unique(1, /*batches=*/1, nullptr); + }); + LazyConcatBatchReader reader(SourcesOf(std::move(factories)), GetDefaultPool()); + + Result first = reader.NextBatch(); + ASSERT_FALSE(first.ok()); + ASSERT_NOK_WITH_MSG(first, "cannot read file-0"); + // The second call answers with the same failure and never reaches the second file. + Result second = reader.NextBatch(); + ASSERT_FALSE(second.ok()); + ASSERT_NOK_WITH_MSG(second, "cannot read file-0"); + ASSERT_TRUE(second.status().IsIOError()); + ASSERT_EQ(opened, 1); +} + +TEST(LazyConcatBatchReaderTest, TestAFailureNamesTheFileItCameFrom) { + // A split holds a whole partition, so "the file ended inside a quoted value" says nothing a + // caller could act on until it says which file. Both halves are covered: the file that would + // not open, and the file that opened and then failed part way through. + std::vector failing_open; + failing_open.push_back([]() -> Result> { + return Status::NotImplemented("the compression cannot be decoded"); + }); + LazyConcatBatchReader open_reader(SourcesOf(std::move(failing_open)), GetDefaultPool()); + Result not_opened = open_reader.NextBatch(); + ASSERT_FALSE(not_opened.ok()); + ASSERT_NOK_WITH_MSG(not_opened, "cannot read file-0"); + ASSERT_NOK_WITH_MSG(not_opened, "the compression cannot be decoded"); + // The original code survives, or a caller that tells "not supported" from "bad input" would + // stop being able to. + ASSERT_TRUE(not_opened.status().IsNotImplemented()); + + std::vector failing_read; + failing_read.push_back([]() -> Result> { + return std::make_unique(0, /*batches=*/1, nullptr); + }); + failing_read.push_back([]() -> Result> { + return std::make_unique(Status::Invalid("the file ended mid-record")); + }); + LazyConcatBatchReader read_reader(SourcesOf(std::move(failing_read)), GetDefaultPool()); + ASSERT_OK_AND_ASSIGN(BatchReader::ReadBatch first, read_reader.NextBatch()); + ASSERT_FALSE(BatchReader::IsEofBatch(first)); + ReaderUtils::ReleaseReadBatch(std::move(first)); + Result failed = read_reader.NextBatch(); + ASSERT_FALSE(failed.ok()); + // The second file, not the first: the name follows the reader in hand. + ASSERT_NOK_WITH_MSG(failed, "cannot read file-1"); + ASSERT_NOK_WITH_MSG(failed, "the file ended mid-record"); + ASSERT_TRUE(failed.status().IsInvalid()); +} + +TEST(LazyConcatBatchReaderTest, TestAFactoryMayDeclineToOpenAnything) { + // A factory that returns no reader stands for a file with nothing to read; the next one is + // reached instead of the read ending there. + std::vector factories; + factories.push_back( + []() -> Result> { return std::unique_ptr(); }); + factories.push_back([]() -> Result> { + return std::make_unique(7, /*batches=*/2, nullptr); + }); + LazyConcatBatchReader reader(SourcesOf(std::move(factories)), GetDefaultPool()); + ASSERT_EQ(DrainRows(&reader), 2); +} + +TEST(LazyConcatBatchReaderTest, TestMetricsCoverReadersThatAreAlreadyClosed) { + std::vector factories; + factories.push_back([]() -> Result> { + return std::make_unique(0, /*batches=*/1, nullptr); + }); + LazyConcatBatchReader reader(SourcesOf(std::move(factories)), GetDefaultPool()); + ASSERT_EQ(DrainRows(&reader), 1); + // The totals have to survive the reader they were taken from, or everything read through a + // file would vanish from them the moment that file is closed. + ASSERT_NE(reader.GetReaderMetrics(), nullptr); +} + +} // namespace paimon::test diff --git a/src/paimon/core/table/sink/commit_message_serializer.cpp b/src/paimon/core/table/sink/commit_message_serializer.cpp index 1b0645f4..1c3778de 100644 --- a/src/paimon/core/table/sink/commit_message_serializer.cpp +++ b/src/paimon/core/table/sink/commit_message_serializer.cpp @@ -39,6 +39,7 @@ #include "paimon/core/io/data_file_meta_first_row_id_legacy_serializer.h" #include "paimon/core/io/data_file_meta_serializer.h" #include "paimon/core/io/data_increment.h" +#include "paimon/core/table/format/format_commit_message.h" #include "paimon/core/table/sink/commit_message_impl.h" #include "paimon/core/utils/object_serializer.h" #include "paimon/io/data_input_stream.h" @@ -57,6 +58,13 @@ CommitMessageSerializer::~CommitMessageSerializer() = default; Status CommitMessageSerializer::Serialize(const std::shared_ptr& obj, MemorySegmentOutputStream* out) { + if (std::dynamic_pointer_cast(obj) != nullptr) { + // A format table's commit message names a staged path rather than files to record in a + // manifest, and has no cross-runtime encoding. + return Status::NotImplemented( + "a FormatCommitMessage cannot be serialized: a format table's write and its commit " + "belong to one process"); + } auto message = std::dynamic_pointer_cast(obj); if (message == nullptr) { return Status::Invalid("failed to cast commit message to commit message impl"); diff --git a/src/paimon/core/table/source/split.cpp b/src/paimon/core/table/source/split.cpp index 007df3cb..5a1d4c97 100644 --- a/src/paimon/core/table/source/split.cpp +++ b/src/paimon/core/table/source/split.cpp @@ -26,6 +26,7 @@ #include "paimon/common/utils/serialization_utils.h" #include "paimon/core/global_index/indexed_split_impl.h" #include "paimon/core/io/data_file_meta_serializer.h" +#include "paimon/core/table/format/format_data_split.h" #include "paimon/core/table/source/data_split_impl.h" #include "paimon/core/table/source/deletion_file.h" #include "paimon/core/table/source/fallback_data_split.h" @@ -165,6 +166,12 @@ Result Split::Serialize(const std::shared_ptr& split, } else { out.WriteValue(false); } + } else if (std::dynamic_pointer_cast(split) != nullptr) { + // A format table's plan has no cross-runtime encoding, and one of paimon-cpp's own would + // let a plan made here be handed to a runtime that cannot read it back. + return Status::NotImplemented( + "a FormatDataSplit cannot be serialized: a format table's plan is in-memory only, so " + "plan and read it within one process"); } else { return Status::Invalid("invalid split, cannot cast to DataSplit or IndexedSplit"); } diff --git a/src/paimon/core/table/source/table_read.cpp b/src/paimon/core/table/source/table_read.cpp index 3d6a3676..5c518e51 100644 --- a/src/paimon/core/table/source/table_read.cpp +++ b/src/paimon/core/table/source/table_read.cpp @@ -21,18 +21,24 @@ #include #include +#include #include #include #include +#include +#include "arrow/c/bridge.h" #include "fmt/format.h" #include "paimon/common/reader/concat_batch_reader.h" #include "paimon/common/types/data_field.h" +#include "paimon/common/utils/arrow/status_utils.h" #include "paimon/common/utils/string_utils.h" #include "paimon/core/core_options.h" #include "paimon/core/operation/internal_read_context.h" #include "paimon/core/schema/schema_manager.h" #include "paimon/core/schema/table_schema.h" +#include "paimon/core/table/format/format_table_loader.h" +#include "paimon/core/table/format/format_table_read.h" #include "paimon/core/table/source/append_only_table_read.h" #include "paimon/core/table/source/fallback_table_read.h" #include "paimon/core/table/source/key_value_table_read.h" @@ -43,6 +49,7 @@ #include "paimon/format/file_format.h" #include "paimon/read_context.h" #include "paimon/status.h" +#include "paimon/table/format/format_table.h" namespace paimon { class DataSplit; @@ -51,12 +58,18 @@ class MemoryPool; namespace { +/// @param loaded_schema The schema of `branch`, when the caller already read it, or null when it +/// has to be read here. `TableRead::Create()` reads one to dispatch on its table type, and +/// reading it again for the branch it came from would cost a second listing and read. Result> CreateInternalReadContext( - const std::shared_ptr& context, const std::string& branch) { + const std::shared_ptr& context, const std::string& branch, + const std::shared_ptr& loaded_schema) { std::map tmp_options = context->GetOptions(); std::shared_ptr table_schema; const auto& specific_table_schema = context->GetSpecificTableSchema(); - if (branch == BranchManager::DEFAULT_MAIN_BRANCH && specific_table_schema) { + if (loaded_schema != nullptr) { + table_schema = loaded_schema; + } else if (branch == BranchManager::DEFAULT_MAIN_BRANCH && specific_table_schema) { PAIMON_ASSIGN_OR_RAISE(table_schema, TableSchema::CreateFromJson(specific_table_schema.value())); } else { @@ -115,9 +128,11 @@ Result> CreateTableRead( return KeyValueTableRead::Create(path_factory, internal_context, memory_pool, executor); } -Result> NewDataTableRead(const std::shared_ptr& context) { +Result> NewDataTableRead( + const std::shared_ptr& context, + const std::shared_ptr& loaded_schema) { PAIMON_ASSIGN_OR_RAISE(std::shared_ptr internal_context, - CreateInternalReadContext(context, context->GetBranch())); + CreateInternalReadContext(context, context->GetBranch(), loaded_schema)); PAIMON_ASSIGN_OR_RAISE( std::unique_ptr table_read, @@ -132,7 +147,8 @@ Result> NewDataTableRead(const std::shared_ptr fallback_context, - CreateInternalReadContext(context, /*branch=*/scan_fallback_branch.value())); + CreateInternalReadContext(context, /*branch=*/scan_fallback_branch.value(), + /*loaded_schema=*/nullptr)); PAIMON_ASSIGN_OR_RAISE( std::unique_ptr fallback_table_read, @@ -145,6 +161,20 @@ Result> NewDataTableRead(const std::shared_ptr& memory_pool) : pool_(memory_pool) {} +namespace { + +/// Maps a `ReadContext` onto `FormatTableRead`, which reads everything it needs out of the context +/// itself: the columns, the predicate, the pool and executor, and what a file is opened with. It +/// refuses by name what a format table cannot honour. +Result> NewFormatTableRead(const std::shared_ptr& table, + const std::shared_ptr& context) { + PAIMON_ASSIGN_OR_RAISE(std::unique_ptr read, + FormatTableRead::Create(table, context)); + return std::unique_ptr(std::move(read)); +} + +} // namespace + Result> TableRead::Create(std::unique_ptr ctx) { std::shared_ptr context = std::move(ctx); if (context == nullptr) { @@ -170,7 +200,24 @@ Result> TableRead::Create(std::unique_ptrNewRead(context); } - return NewDataTableRead(context); + // A format table has no manifest to read its files through, and a file another engine wrote + // carries no field ids, so which files there are and how a row is put back together is its + // own. How a file is opened is not: both paths go through `DataFileReaderFactory`, so prefetch + // and the cache apply either way. One `TableRead` interface serves both, as Java Paimon serves + // both through one `ReadBuilder`. + std::shared_ptr latest_schema; + PAIMON_ASSIGN_OR_RAISE( + std::shared_ptr format_table, + FormatTableLoader::TryLoad(tmp_core_options.GetFileSystem(), context->GetPath(), + context->GetBranch(), context->GetOptions(), + context->GetSpecificTableSchema(), + /*schema_manager=*/nullptr, &latest_schema)); + if (format_table != nullptr) { + return NewFormatTableRead(format_table, context); + } + + // With the schema the dispatch already read, so the managed path does not read it again. + return NewDataTableRead(context, latest_schema); } Result> TableRead::CreateReader( diff --git a/src/paimon/core/table/source/table_scan.cpp b/src/paimon/core/table/source/table_scan.cpp index 2dda955a..11f0e74f 100644 --- a/src/paimon/core/table/source/table_scan.cpp +++ b/src/paimon/core/table/source/table_scan.cpp @@ -46,6 +46,8 @@ #include "paimon/core/schema/schema_validation.h" #include "paimon/core/schema/table_schema.h" #include "paimon/core/table/bucket_mode.h" +#include "paimon/core/table/format/format_table_loader.h" +#include "paimon/core/table/format/format_table_scan.h" #include "paimon/core/table/source/abstract_table_scan.h" #include "paimon/core/table/source/append_only_split_generator.h" #include "paimon/core/table/source/data_evolution_batch_scan.h" @@ -69,6 +71,7 @@ #include "paimon/result.h" #include "paimon/scan_context.h" #include "paimon/status.h" +#include "paimon/table/format/format_table.h" namespace arrow { class Schema; @@ -183,10 +186,73 @@ class TableScanImpl { } }; -Result> NewDataTableScan(const std::shared_ptr& context); +/// @param loaded_schema The table's schema, when the caller already read it, or null when it has +/// to be read here. `TableScan::Create()` reads one to dispatch on its table type. +Result> NewDataTableScan( + const std::shared_ptr& context, const std::shared_ptr& loaded_schema); } // namespace +namespace { + +/// Maps a `ScanContext` onto `FormatTableScan`, which takes one partition filter and a limit and +/// nothing else. +/// +/// A format table has no buckets, so a bucket filter has nothing to act on. A predicate is refused +/// too, which is narrower than Java: `FormatTableScan.withFilter` refuses one there as well, but +/// `FormatReadBuilder.newScan()` splits a filter first and hands the partition half to the scan, +/// so a predicate over partition columns prunes directories. Doing the same needs the scan to +/// evaluate a predicate against a partition read out of the directory names rather than to match +/// exact values, which it does not do yet. +Result> NewFormatTableScan(const std::shared_ptr& table, + const std::shared_ptr& context) { + // Anything a `ScanContext` carries that a format table cannot honour is refused rather than + // silently dropped. + if (context->IsStreamingMode()) { + return Status::NotImplemented( + "a format table has no snapshots, so there is nothing for a streaming scan to follow"); + } + if (context->GetRealtimeContext() != nullptr) { + return Status::NotImplemented( + "a format table has no real-time store to union with what is on disk"); + } + if (context->GetGlobalIndexResult() != nullptr) { + return Status::NotImplemented("a format table carries no global index"); + } + std::map partition_filter; + std::shared_ptr filters = context->GetScanFilters(); + if (filters != nullptr) { + if (filters->GetPredicate() != nullptr) { + return Status::NotImplemented( + "a format table scan does not take a predicate: its files carry no statistics to " + "skip by, and pruning by the partition columns a predicate names is not " + "implemented yet; give the partition values as a partition filter instead"); + } + if (filters->GetBucketFilter()) { + return Status::NotImplemented("a format table has no buckets to filter by"); + } + const std::vector>& partition_filters = + filters->GetPartitionFilters(); + if (partition_filters.size() > 1) { + return Status::NotImplemented("a format table scan takes at most one partition filter"); + } + if (partition_filters.size() == 1) { + partition_filter = partition_filters.front(); + } + } + PAIMON_ASSIGN_OR_RAISE(std::unique_ptr scan, + FormatTableScan::Create(table, partition_filter, context->GetLimit())); + return std::unique_ptr(std::move(scan)); +} + +} // namespace + +Result>> TableScan::ListPartitions() const { + // A managed table's partitions live in its manifests, and reading them is a scan of its own + // rather than a listing, so only the format table path answers this for now. + return Status::NotImplemented("this table does not list its partitions"); +} + Result> TableScan::Create(std::unique_ptr context) { if (context == nullptr) { return Status::Invalid("scan context is null pointer"); @@ -212,7 +278,22 @@ Result> TableScan::Create(std::unique_ptrGetOptions())); return system_table->NewScan(shared_context); } - return NewDataTableScan(shared_context); + // A format table is planned by listing directories, so it never reaches the manifest path + // below. One `TableScan` interface serves both, as Java Paimon serves both through one + // `ReadBuilder`. + std::shared_ptr latest_schema; + PAIMON_ASSIGN_OR_RAISE( + std::shared_ptr format_table, + FormatTableLoader::TryLoad(tmp_options.GetFileSystem(), shared_context->GetPath(), + BranchManager::NormalizeBranch(tmp_options.GetBranch()), + shared_context->GetOptions(), + shared_context->GetSpecificTableSchema(), + /*schema_manager=*/nullptr, &latest_schema)); + if (format_table != nullptr) { + return NewFormatTableScan(format_table, shared_context); + } + // With the schema the dispatch already read, so the managed path does not read it again. + return NewDataTableScan(shared_context, latest_schema); } namespace { @@ -250,14 +331,18 @@ Status ValidateRealtimeScan(const TableSchema& table_schema, const CoreOptions& return Status::OK(); } -Result> NewDataTableScan(const std::shared_ptr& context) { +Result> NewDataTableScan( + const std::shared_ptr& context, + const std::shared_ptr& loaded_schema) { PAIMON_ASSIGN_OR_RAISE( CoreOptions tmp_options, CoreOptions::FromMap(context->GetOptions(), context->GetSpecificFileSystem(), {})); std::string branch = BranchManager::NormalizeBranch(tmp_options.GetBranch()); std::shared_ptr table_schema; const auto& specific_table_schema = context->GetSpecificTableSchema(); - if (branch == BranchManager::DEFAULT_MAIN_BRANCH && specific_table_schema) { + if (loaded_schema != nullptr) { + table_schema = loaded_schema; + } else if (branch == BranchManager::DEFAULT_MAIN_BRANCH && specific_table_schema) { PAIMON_ASSIGN_OR_RAISE(table_schema, TableSchema::CreateFromJson(specific_table_schema.value())); } else { diff --git a/src/paimon/core/table/table.cpp b/src/paimon/core/table/table.cpp index f0c1de2f..eaa57440 100644 --- a/src/paimon/core/table/table.cpp +++ b/src/paimon/core/table/table.cpp @@ -22,6 +22,7 @@ #include "fmt/format.h" #include "paimon/common/utils/checked_cast.h" +#include "paimon/core/catalog/catalog_utils.h" #include "paimon/core/schema/schema_manager.h" #include "paimon/core/schema/table_schema.h" #include "paimon/fs/file_system.h" @@ -44,6 +45,12 @@ Result> Table::Create(const std::shared_ptr& fmt::format("load table schema for {} failed", identifier.ToString())); } + // Only a managed table is a `Table`; describing another type as one would hand back snapshots + // and manifests it never had. Checked here because this is the one place every `Table` built + // from a table directory passes through, catalogs included. + PAIMON_RETURN_NOT_OK( + CatalogUtils::CheckManagedTableType(identifier, *latest_schema, "Table::Create")); + auto schema = checked_pointer_cast(*latest_schema); return std::make_shared
(schema, identifier.GetDatabaseName(), identifier.GetTableName()); } diff --git a/src/paimon/core/table/table_test.cpp b/src/paimon/core/table/table_test.cpp index f2d12f25..60113ec8 100644 --- a/src/paimon/core/table/table_test.cpp +++ b/src/paimon/core/table/table_test.cpp @@ -18,6 +18,11 @@ #include "paimon/catalog/table.h" +#include +#include +#include +#include + #include "arrow/api.h" #include "gtest/gtest.h" #include "paimon/core/schema/schema_manager.h" @@ -61,6 +66,35 @@ TEST(TableTest, TestCreateWithUnknownDatabase) { EXPECT_EQ(data_schema->PrimaryKeys(), primary_keys); } +TEST(TableTest, TestCreateRejectsATableTypeThatIsNotAManagedTable) { + // A `Table` describes a managed paimon table. Handing back another type as one would promise + // snapshots and manifests it never had, so the failure belongs at the call that opens the + // table rather than at the first read. This is the one place every `Table` built from a table + // directory passes through, catalogs included. + auto dir = UniqueTestDirectory::Create(); + ASSERT_TRUE(dir); + auto schema = arrow::schema({arrow::field("id", arrow::int32(), /*nullable=*/false), + arrow::field("dt", arrow::utf8())}); + + SchemaManager format_schema_manager(dir->GetFileSystem(), dir->Str() + "/format"); + ASSERT_OK_AND_ASSIGN( + [[maybe_unused]] std::unique_ptr format_schema, + format_schema_manager.CreateTable(schema, /*partition_keys=*/{"dt"}, + /*primary_keys=*/{}, + {{"type", "format-table"}, {"file.format", "parquet"}})); + ASSERT_NOK_WITH_MSG( + Table::Create(dir->GetFileSystem(), dir->Str() + "/format", Identifier("db", "fmt")), + "Cannot open format table"); + + // A managed table with no `type` option at all is what a `Table` is for, and still opens. + SchemaManager managed_schema_manager(dir->GetFileSystem(), dir->Str() + "/managed"); + ASSERT_OK_AND_ASSIGN([[maybe_unused]] std::unique_ptr managed_schema, + managed_schema_manager.CreateTable(schema, /*partition_keys=*/{"dt"}, + /*primary_keys=*/{}, {})); + ASSERT_OK( + Table::Create(dir->GetFileSystem(), dir->Str() + "/managed", Identifier("db", "managed"))); +} + TEST(TableTest, TestCreateFailedWithNonExistSchema) { auto dir = UniqueTestDirectory::Create(); ASSERT_TRUE(dir); diff --git a/src/paimon/core/utils/partition_path_utils.cpp b/src/paimon/core/utils/partition_path_utils.cpp index 632feb84..303f77c8 100644 --- a/src/paimon/core/utils/partition_path_utils.cpp +++ b/src/paimon/core/utils/partition_path_utils.cpp @@ -22,6 +22,7 @@ #include #include #include +#include #include "paimon/status.h" @@ -49,8 +50,17 @@ const std::bitset<128>& PartitionPathUtils::CharToEscape() { return bitset; } +Status PartitionPathUtils::ValidatePartitionValueForPath(const std::string& value, + bool only_value) { + if (value.empty() || (only_value && (value == "." || value == ".."))) { + return Status::Invalid("Partition value '" + value + + "' cannot be used as a partition path component."); + } + return Status::OK(); +} + Result PartitionPathUtils::GeneratePartitionPath( - const std::vector>& partition_spec) { + const std::vector>& partition_spec, bool only_value) { if (partition_spec.empty()) { return std::string(); } @@ -60,9 +70,13 @@ Result PartitionPathUtils::GeneratePartitionPath( if (i > 0) { ss << PATH_SEPARATOR; } - PAIMON_ASSIGN_OR_RAISE(std::string key_esc, EscapePathName(key)); + if (!only_value) { + PAIMON_ASSIGN_OR_RAISE(std::string key_esc, EscapePathName(key)); + ss << key_esc << "="; + } + PAIMON_RETURN_NOT_OK(ValidatePartitionValueForPath(value, only_value)); PAIMON_ASSIGN_OR_RAISE(std::string value_esc, EscapePathName(value)); - ss << key_esc << "=" << value_esc; + ss << value_esc; i++; } ss << PATH_SEPARATOR; @@ -95,6 +109,62 @@ Result PartitionPathUtils::EscapePathName(const std::string& path) return ss.value().str(); } +namespace { +/// Value of one hexadecimal digit, or -1 when `c` is not one. +/// +/// Written out rather than handed to `strtol`, which also accepts a leading sign or whitespace and +/// so would read `"% 1"` and `"%+1"` as escape sequences that `EscapePathName` never produces. +int32_t HexDigit(char c) { + if (c >= '0' && c <= '9') { + return c - '0'; + } + if (c >= 'a' && c <= 'f') { + return c - 'a' + 10; + } + if (c >= 'A' && c <= 'F') { + return c - 'A' + 10; + } + return -1; +} +} // namespace + +std::string PartitionPathUtils::UnescapePathName(const std::string& path) { + std::string result; + result.reserve(path.size()); + for (size_t i = 0; i < path.size(); i++) { + // Not an off-by-one: a `%` among the last two characters starts no sequence and is left + // as written. + if (path[i] == '%' && i + 2 < path.size()) { + const int32_t high = HexDigit(path[i + 1]); + const int32_t low = HexDigit(path[i + 2]); + if (high >= 0 && low >= 0) { + result.push_back(static_cast(high * 16 + low)); + i += 2; + continue; + } + } + result.push_back(path[i]); + } + return result; +} + +std::optional> PartitionPathUtils::ExtractPartitionKeyValue( + const std::string& directory_name) { + size_t separator = directory_name.find('='); + // Both halves must be non-empty: `=v` names no key and `k=` no value, and neither is a + // partition directory this table wrote. + if (separator == std::string::npos || separator == 0 || + separator + 1 == directory_name.size()) { + return std::nullopt; + } + // A second `=` cannot appear unescaped, so the name belongs to something else. + if (directory_name.find('=', separator + 1) != std::string::npos) { + return std::nullopt; + } + return std::make_pair(UnescapePathName(directory_name.substr(0, separator)), + UnescapePathName(directory_name.substr(separator + 1))); +} + void PartitionPathUtils::EscapeChar(char c, std::stringstream* ss_ptr) { auto& ss = *ss_ptr; ss << '%'; diff --git a/src/paimon/core/utils/partition_path_utils.h b/src/paimon/core/utils/partition_path_utils.h index 48d9adf3..b0260548 100644 --- a/src/paimon/core/utils/partition_path_utils.h +++ b/src/paimon/core/utils/partition_path_utils.h @@ -23,12 +23,14 @@ #include #include #include +#include #include #include #include #include #include "paimon/result.h" +#include "paimon/status.h" namespace paimon { @@ -42,9 +44,13 @@ class PartitionPathUtils { /// Make partition path from partition spec. /// /// @param partition_spec The partition spec. + /// @param only_value Name each level by its escaped value alone (`2025/01/`) instead of + /// `key=value` (`year=2025/month=01/`). That is the layout a format table takes when + /// `format-table.partition-path-only-value` is on. /// @return An escaped, valid partition name. static Result GeneratePartitionPath( - const std::vector>& partition_spec); + const std::vector>& partition_spec, + bool only_value = false); /// Escapes a path name. /// @@ -52,6 +58,34 @@ class PartitionPathUtils { /// @return An escaped path name. static Result EscapePathName(const std::string& path); + /// Reverses `EscapePathName`, turning every `%XX` sequence back into the character it stands + /// for. A `%` that does not start a valid sequence is kept as written. + static std::string UnescapePathName(const std::string& path); + + /// Splits a `key=value` partition directory name into its unescaped key and value. + /// + /// Returns nullopt when the name is not of that shape, which is how a directory that is not a + /// partition of this table is told apart from one that is. + /// + /// A name carrying a second unescaped `=` is not of that shape: `EscapePathName` escapes + /// `=`, so no directory paimon wrote looks like that, and one written by something else is + /// skipped rather than bound to a key it does not name. + static std::optional> ExtractPartitionKeyValue( + const std::string& directory_name); + + /// Whether a path component is hidden by the `_` / `.` convention every engine writing a + /// Hive-style directory uses to mark output that is not committed table data. + static bool IsHiddenName(const std::string& name) { + return !name.empty() && (name[0] == '_' || name[0] == '.'); + } + + /// Fails when `value` cannot name a partition directory. + /// + /// A value-only directory is the bare value, so "." and ".." would name the directory itself + /// and its parent instead of a partition; under `key=value` the "=" already keeps them apart + /// from a relative path. + static Status ValidatePartitionValueForPath(const std::string& value, bool only_value); + /// Generate all hierarchical paths from partition spec. /// /// For example, if the partition spec is (pt1: '0601', pt2: '12', pt3: '30'), this method diff --git a/src/paimon/core/utils/partition_path_utils_test.cpp b/src/paimon/core/utils/partition_path_utils_test.cpp index a10729b6..7444a191 100644 --- a/src/paimon/core/utils/partition_path_utils_test.cpp +++ b/src/paimon/core/utils/partition_path_utils_test.cpp @@ -103,4 +103,88 @@ TEST(PartitionPathUtilsTest, EscapePathName) { ASSERT_EQ(escape_path, " %2F%3D"); } +TEST(PartitionPathUtilsTest, UnescapePathName) { + ASSERT_EQ(PartitionPathUtils::UnescapePathName(""), ""); + ASSERT_EQ(PartitionPathUtils::UnescapePathName("normal_path"), "normal_path"); + // Round trips whatever `EscapePathName` produced, in either digit case. + ASSERT_EQ(PartitionPathUtils::UnescapePathName("a b%2Fc"), "a b/c"); + ASSERT_EQ(PartitionPathUtils::UnescapePathName("a b%2fc"), "a b/c"); + ASSERT_EQ(PartitionPathUtils::UnescapePathName("%25"), "%"); + + // A `%` that starts no sequence stays as written, which is what lets a directory another + // engine wrote be read at all. + ASSERT_EQ(PartitionPathUtils::UnescapePathName("100%"), "100%"); + ASSERT_EQ(PartitionPathUtils::UnescapePathName("%zz"), "%zz"); + ASSERT_EQ(PartitionPathUtils::UnescapePathName("%2"), "%2"); + // Only two plain hex digits count: neither whitespace nor a sign starts a sequence, and + // `EscapePathName` never writes one. + ASSERT_EQ(PartitionPathUtils::UnescapePathName("% 1x"), "% 1x"); + ASSERT_EQ(PartitionPathUtils::UnescapePathName("%+1x"), "%+1x"); + // A `%` among the last two characters starts no sequence. + ASSERT_EQ(PartitionPathUtils::UnescapePathName("a%41"), "aA"); + ASSERT_EQ(PartitionPathUtils::UnescapePathName("%41"), "A"); +} + +TEST(PartitionPathUtilsTest, ExtractPartitionKeyValue) { + std::optional> key_value = + PartitionPathUtils::ExtractPartitionKeyValue("dt=2025-01-01"); + ASSERT_TRUE(key_value.has_value()); + ASSERT_EQ(key_value->first, "dt"); + ASSERT_EQ(key_value->second, "2025-01-01"); + + // Both halves are unescaped, so a value holding an escaped `=` reads back whole. + key_value = PartitionPathUtils::ExtractPartitionKeyValue("dt=a%3Db"); + ASSERT_TRUE(key_value.has_value()); + ASSERT_EQ(key_value->second, "a=b"); + + // Anything that is not one `key=value` belongs to something else: another layout, a nested + // table, or a directory this table never wrote. + ASSERT_FALSE(PartitionPathUtils::ExtractPartitionKeyValue("dt").has_value()); + ASSERT_FALSE(PartitionPathUtils::ExtractPartitionKeyValue("=2025").has_value()); + ASSERT_FALSE(PartitionPathUtils::ExtractPartitionKeyValue("dt=").has_value()); + // A `=` is escaped on the way in, so no directory paimon wrote carries a second one. + ASSERT_FALSE(PartitionPathUtils::ExtractPartitionKeyValue("dt=a=b").has_value()); +} + +TEST(PartitionPathUtilsTest, IsHiddenName) { + ASSERT_TRUE(PartitionPathUtils::IsHiddenName("_temporary")); + ASSERT_TRUE(PartitionPathUtils::IsHiddenName(".hive-staging_1")); + ASSERT_FALSE(PartitionPathUtils::IsHiddenName("dt=2025-01-01")); + ASSERT_FALSE(PartitionPathUtils::IsHiddenName("")); +} + +TEST(PartitionPathUtilsTest, ValidatePartitionValueForPath) { + ASSERT_OK(PartitionPathUtils::ValidatePartitionValueForPath("2025", /*only_value=*/false)); + ASSERT_OK(PartitionPathUtils::ValidatePartitionValueForPath("2025", /*only_value=*/true)); + + // An empty value names no directory under either layout. + ASSERT_NOK_WITH_MSG(PartitionPathUtils::ValidatePartitionValueForPath("", false), + "cannot be used as a partition path component"); + // A bare "." or ".." would name the directory itself or its parent; under `key=value` the + // "=" already keeps them apart from a relative path. + ASSERT_NOK(PartitionPathUtils::ValidatePartitionValueForPath(".", true)); + ASSERT_NOK(PartitionPathUtils::ValidatePartitionValueForPath("..", true)); + ASSERT_OK(PartitionPathUtils::ValidatePartitionValueForPath("..", false)); +} + +TEST(PartitionPathUtilsTest, GenerateValueOnlyPartitionPath) { + std::vector> partition_spec = { + {"year", "2025"}, + {"month", "01"}, + }; + ASSERT_OK_AND_ASSIGN(std::string partition_path, PartitionPathUtils::GeneratePartitionPath( + partition_spec, /*only_value=*/true)); + ASSERT_EQ(partition_path, "2025/01/"); + + // The value is still escaped, so a separator inside it cannot add a level. + ASSERT_OK_AND_ASSIGN(partition_path, PartitionPathUtils::GeneratePartitionPath( + {{"dt", "a/b"}}, /*only_value=*/true)); + ASSERT_EQ(partition_path, "a%2Fb/"); + + // A value the layout cannot name is refused where the path is built, which is the one place + // every writer and every commit goes through. + ASSERT_NOK(PartitionPathUtils::GeneratePartitionPath({{"dt", ".."}}, /*only_value=*/true)); + ASSERT_NOK(PartitionPathUtils::GeneratePartitionPath({{"dt", ""}}, /*only_value=*/false)); +} + } // namespace paimon::test diff --git a/src/paimon/rest/rest_catalog.cpp b/src/paimon/rest/rest_catalog.cpp index 5d2bb823..dcfc292e 100644 --- a/src/paimon/rest/rest_catalog.cpp +++ b/src/paimon/rest/rest_catalog.cpp @@ -29,6 +29,7 @@ #include "paimon/common/utils/string_utils.h" #include "paimon/core/catalog/catalog_utils.h" #include "paimon/core/core_options.h" +#include "paimon/core/schema/schema_validation.h" #include "paimon/core/schema/table_schema.h" #include "paimon/core/table/system/global_system_tables.h" #include "paimon/core/table/system/system_table.h" @@ -36,6 +37,7 @@ #include "paimon/defs.h" #include "paimon/fs/file_system.h" #include "paimon/rest/rest_util.h" +#include "paimon/table/format/format_table.h" #include "rapidjson/document.h" namespace paimon { @@ -194,6 +196,10 @@ Status RestCatalog::CreateTable(const Identifier& identifier, ArrowSchema* c_sch PAIMON_ASSIGN_OR_RAISE(std::unique_ptr table_schema, TableSchema::Create(TableSchema::FIRST_SCHEMA_ID, schema, partition_keys, primary_keys, effective_options)); + // The same checks the file system catalog runs, on the options this will actually send: the + // server takes schemas this library cannot open. + PAIMON_RETURN_NOT_OK(SchemaValidation::ValidateNewTableSchema(*table_schema)); + std::string schema_json; try { rapidjson::Document doc; @@ -404,8 +410,28 @@ Result> RestCatalog::LoadTableSchema(const Identifier& i return checked_pointer_cast(schema); } +Result> RestCatalog::LoadFormatTable( + const Identifier& identifier) const { + PAIMON_ASSIGN_OR_RAISE(bool is_system_table, identifier.IsSystemTable()); + if (is_system_table || CatalogUtils::IsSystemDatabase(identifier.GetDatabaseName())) { + return Status::Invalid(fmt::format("{} is a system table, so it cannot be a format table", + identifier.GetFullName())); + } + PAIMON_ASSIGN_OR_RAISE(std::optional branch, identifier.GetBranchName()); + branch = NormalizeBranch(std::move(branch)); + PAIMON_ASSIGN_OR_RAISE(Identifier load_identifier, ToLoadIdentifier(identifier)); + std::string location; + PAIMON_ASSIGN_OR_RAISE(std::shared_ptr schema, + LoadDataTableSchema(load_identifier, branch, &location)); + // A rest catalog holds the schema itself, so everything below the location is data. + return FormatTable::Create(fs_, location, identifier, checked_pointer_cast(schema), + /*location_carries_paimon_metadata=*/false); +} + Result> RestCatalog::GetTable(const Identifier& identifier) const { PAIMON_ASSIGN_OR_RAISE(std::shared_ptr schema, LoadTableSchema(identifier)); + PAIMON_RETURN_NOT_OK( + CatalogUtils::CheckManagedTableType(identifier, schema, "Catalog::GetTable")); return std::make_shared
(schema, identifier.GetDatabaseName(), identifier.GetTableName()); } diff --git a/src/paimon/rest/rest_catalog.h b/src/paimon/rest/rest_catalog.h index 984aad70..54d60caf 100644 --- a/src/paimon/rest/rest_catalog.h +++ b/src/paimon/rest/rest_catalog.h @@ -76,6 +76,12 @@ class RestCatalog : public Catalog { /// Options merged with the server side config. const std::map& GetOptions() const override; + protected: + /// Loads the location and the schema from one `GetTable` response, so the table cannot be + /// built from a location and a schema that two requests disagreed on. + Result> LoadFormatTable( + const Identifier& identifier) const override; + private: RestCatalog(std::unique_ptr api, const std::shared_ptr& fs, const std::string& warehouse); diff --git a/src/paimon/rest/rest_catalog_test.cpp b/src/paimon/rest/rest_catalog_test.cpp index fda03cc4..e351e209 100644 --- a/src/paimon/rest/rest_catalog_test.cpp +++ b/src/paimon/rest/rest_catalog_test.cpp @@ -41,6 +41,7 @@ #include "paimon/rest/mock_rest_server.h" #include "paimon/rest/rest_api.h" #include "paimon/schema/schema.h" +#include "paimon/table/format/format_table.h" #include "paimon/testing/utils/testharness.h" namespace paimon::test { @@ -63,6 +64,12 @@ struct MockCatalogState { std::map last_headers; // when set, every request except "/v1/config" fails with this http code std::optional force_error_code; + // how many times a single table has been fetched, so a caller that needs the path and the + // schema together can be held to one round trip + int32_t get_table_requests = 0; + // how many create-table requests reached the server, so a schema the client should have + // refused can be shown never to have been sent + int32_t create_table_requests = 0; // guards all fields above: the handler runs on the server's accept thread while // tests seed and inspect the state std::mutex mutex; @@ -248,6 +255,7 @@ MockRestServer::Response HandleCatalogRequest(MockCatalogState* state, return JsonResponse(200, response.ToJsonString().value()); } if (request.method == "POST") { + state->create_table_requests++; CreateTableRequest create_request("", "", ""); if (!RapidJsonUtil::FromJsonString(request.body, &create_request).ok()) { return MockError(400, "", "", "bad create table request"); @@ -290,6 +298,7 @@ MockRestServer::Response HandleCatalogRequest(MockCatalogState* state, return JsonResponse(200, fmt::format(R"({{"snapshots":[{}]}})", SnapshotJson(1))); } if (request.method == "GET") { + state->get_table_requests++; return JsonResponse(200, TableResponseJson(table_name, table_iter->second)); } if (request.method == "DELETE") { @@ -678,6 +687,204 @@ TEST_F(RestCatalogTest, BranchTableLoadsBranchSchemaFromServer) { "branch table"); } +TEST_F(RestCatalogTest, CreateRefusesAFormatTableThisClientCouldNotOpen) { + ASSERT_OK_AND_ASSIGN(std::unique_ptr catalog, CreateRestCatalog()); + ASSERT_OK(catalog->CreateDatabase("db1", {}, /*ignore_if_exists=*/false)); + + // The server takes schemas this library cannot open. Without the same checks the file system + // catalog runs, creating through this client would succeed and opening the very same table + // through it would fail. + auto create = [&catalog](const std::string& name, const std::shared_ptr& schema, + const std::vector& partition_keys, + const std::map& options) { + struct ArrowSchema c_schema; + EXPECT_TRUE(arrow::ExportSchema(*schema, &c_schema).ok()); + Status status = catalog->CreateTable(Identifier("db1", name), &c_schema, partition_keys, + /*primary_keys=*/{}, options, false); + if (c_schema.release != nullptr) { + c_schema.release(&c_schema); + } + return status; + }; + const std::map format_table = {{Options::TYPE, "format-table"}, + {Options::FILE_FORMAT, "parquet"}}; + + { + std::lock_guard lock(state_->mutex); + state_->create_table_requests = 0; + } + + // Every column a partition column leaves the data files with nothing in them. + std::shared_ptr only_partitions = + arrow::schema({arrow::field("dt", arrow::utf8())}); + ASSERT_NOK_WITH_MSG(create("only_partitions", only_partitions, {"dt"}, format_table), + "every one of its columns"); + + // A partition type this library does not support. + std::shared_ptr timestamp_partition = + arrow::schema({arrow::field("f0", arrow::int32()), + arrow::field("dt", arrow::timestamp(arrow::TimeUnit::MICRO))}); + ASSERT_NOK_WITH_MSG(create("timestamp_partition", timestamp_partition, {"dt"}, format_table), + "cannot be TIMESTAMP/DECIMAL/BLOB"); + + // A format table format with no reader here. + std::shared_ptr good = + arrow::schema({arrow::field("f0", arrow::int32()), arrow::field("dt", arrow::utf8())}); + std::map csv_table = format_table; + csv_table[Options::FILE_FORMAT] = "csv"; + ASSERT_NOK_WITH_MSG(create("csv_table", good, {"dt"}, csv_table), + "not supported by paimon-cpp yet"); + + // None of them was sent: the table must not exist on the server for another client to find. + { + std::lock_guard lock(state_->mutex); + ASSERT_EQ(0, state_->create_table_requests); + } + + // And a schema this library can open is created and opens. + ASSERT_OK(create("fine", good, {"dt"}, format_table)); + ASSERT_OK_AND_ASSIGN(std::shared_ptr table, + catalog->GetFormatTable(Identifier("db1", "fine"))); + ASSERT_EQ(FormatTable::Format::PARQUET, table->GetFormat()); +} + +TEST_F(RestCatalogTest, TableResponseWithoutAPathIsRejected) { + ASSERT_OK_AND_ASSIGN(std::unique_ptr catalog, CreateRestCatalog()); + ASSERT_OK(catalog->CreateDatabase("db1", {}, /*ignore_if_exists=*/false)); + + MockCatalogState::TableData no_path; + no_path.schema_json = R"({"fields": [{"id": 0, "name": "f0", "type": "INT NOT NULL"}],)" + R"( "partitionKeys": [], "primaryKeys": [],)" + R"( "options": {"type": "format-table", "file.format": "parquet"}})"; + no_path.path = ""; + { + std::lock_guard lock(state_->mutex); + state_->databases["db1"]["no_path"] = no_path; + } + + // Everything built from the response reads and writes below the path. A table whose paths + // would be checked against an empty one has no boundary at all, so the response is refused + // rather than turned into a table. What the response said is not repeated back: a body may + // carry credentials, so a failure to read one names the request and nothing else. + ASSERT_NOK_WITH_MSG(catalog->GetFormatTable(Identifier("db1", "no_path")), + "failed to deserialize the response"); + ASSERT_NOK_WITH_MSG(catalog->GetTableLocation(Identifier("db1", "no_path")), + "failed to deserialize the response"); +} + +TEST_F(RestCatalogTest, FormatTableIsLoadedInOneRequest) { + ASSERT_OK_AND_ASSIGN(std::unique_ptr catalog, CreateRestCatalog()); + ASSERT_OK(catalog->CreateDatabase("db1", {}, /*ignore_if_exists=*/false)); + + MockCatalogState::TableData table_data; + table_data.schema_json = R"({"fields": [{"id": 0, "name": "f0", "type": "INT NOT NULL"},)" + R"( {"id": 1, "name": "dt", "type": "STRING"}],)" + R"( "partitionKeys": ["dt"], "primaryKeys": [],)" + R"( "options": {"type": "format-table", "file.format": "parquet"}})"; + table_data.schema_id = 7; + table_data.path = "wh1/db1.db/fmt"; + { + std::lock_guard lock(state_->mutex); + state_->databases["db1"]["fmt"] = table_data; + state_->get_table_requests = 0; + } + + ASSERT_OK_AND_ASSIGN(std::shared_ptr table, + catalog->GetFormatTable(Identifier("db1", "fmt"))); + ASSERT_EQ("wh1/db1.db/fmt", table->Location()); + ASSERT_EQ(FormatTable::Format::PARQUET, table->GetFormat()); + ASSERT_EQ((std::vector{"dt"}), table->PartitionKeys()); + // The location and the schema come from one response, so they cannot describe two different + // states of the table. + { + std::lock_guard lock(state_->mutex); + ASSERT_EQ(1, state_->get_table_requests); + } + + // This catalog holds the schema itself, so everything below the location is data - a + // directory named "schema" there is a partition value, not metadata to skip. + ASSERT_FALSE(table->LocationCarriesPaimonMetadata()); + + // And the other way round: a `Table` promises snapshots and manifests a format table never + // had, so it is refused there in the same words the file system catalog uses. + ASSERT_NOK_WITH_MSG(catalog->GetTable(Identifier("db1", "fmt")), "Cannot open format table"); + + // A managed table stays out of this path, and so does a system table. + ASSERT_OK(CreateSampleTable(catalog.get(), Identifier("db1", "t1"))); + ASSERT_NOK_WITH_MSG(catalog->GetFormatTable(Identifier("db1", "t1")), "is not a format table"); + ASSERT_NOK_WITH_MSG(catalog->GetFormatTable(Identifier("db1", "t1$snapshots")), + "is a system table"); +} + +TEST_F(RestCatalogTest, FormatTableWithAnUnusableSchemaIsRejectedOnLoad) { + ASSERT_OK_AND_ASSIGN(std::unique_ptr catalog, CreateRestCatalog()); + ASSERT_OK(catalog->CreateDatabase("db1", {}, /*ignore_if_exists=*/false)); + + // A schema from a rest catalog never passed through table creation in this library, so the + // checks that run there have to run again on load. Without that these would be accepted here + // and fail only when the first reader or writer was built. + const std::string default_fields = R"([{"id": 0, "name": "f0", "type": "INT NOT NULL"},)" + R"( {"id": 1, "name": "dt", "type": "STRING"}])"; + auto seed = [this](const std::string& name, const std::string& options, + const std::string& partition_keys, const std::string& fields) { + MockCatalogState::TableData table_data; + table_data.schema_json = R"({"fields": )" + fields + R"(, "partitionKeys": )" + + partition_keys + R"(, "primaryKeys": [], "options": )" + options + + "}"; + table_data.path = "wh1/db1.db/" + name; + std::lock_guard lock(state_->mutex); + state_->databases["db1"][name] = table_data; + }; + + // A format table format with no reader here. + seed("csv_table", R"({"type": "format-table", "file.format": "csv"})", R"(["dt"])", + default_fields); + ASSERT_NOK_WITH_MSG(catalog->GetFormatTable(Identifier("db1", "csv_table")), + "not supported by paimon-cpp yet"); + + // A row count no file could ever reach. + seed("bad_rows", + R"({"type": "format-table", "file.format": "parquet", "target-file-row-num": "0"})", + R"(["dt"])", default_fields); + ASSERT_NOK_WITH_MSG(catalog->GetFormatTable(Identifier("db1", "bad_rows")), + "should be at least 1"); + + // The failure names the table, which is all a caller holding several has to go on. + ASSERT_NOK_WITH_MSG(catalog->GetFormatTable(Identifier("db1", "bad_rows")), "db1.bad_rows"); + + // The structural rules run here too, not only the format-table ones: a partition key that is + // not a field of the schema is a table nothing could ever read. + seed("bad_partition_key", R"({"type": "format-table", "file.format": "parquet"})", + R"(["nosuchfield"])", default_fields); + ASSERT_NOK_WITH_MSG(catalog->GetFormatTable(Identifier("db1", "bad_partition_key")), + "should include all partition fields"); + + // Two schemas this library cannot open, refused when the table is opened rather than when a + // reader is first built. Both are in the user guide's list of limits. + seed("timestamp_partition", R"({"type": "format-table", "file.format": "parquet"})", + R"(["dt"])", + R"([{"id": 0, "name": "f0", "type": "INT NOT NULL"},)" + // Not a raw string: `TIMESTAMP(6)"` holds the sequence that would end one. + " {\"id\": 1, \"name\": \"dt\", \"type\": \"TIMESTAMP(6)\"}]"); + ASSERT_NOK_WITH_MSG(catalog->GetFormatTable(Identifier("db1", "timestamp_partition")), + "cannot be TIMESTAMP/DECIMAL/BLOB"); + + seed("only_partitions", R"({"type": "format-table", "file.format": "parquet"})", R"(["dt"])", + R"([{"id": 0, "name": "dt", "type": "STRING"}])"); + ASSERT_NOK_WITH_MSG(catalog->GetFormatTable(Identifier("db1", "only_partitions")), + "every one of its columns"); + + // An option this library does not honour keeps its own status code, so a caller can still + // tell "not supported yet" from "bad table". + seed("catalog_partitions", + R"({"type": "format-table", "file.format": "parquet",)" + R"( "metastore.partitioned-table": "true"})", + R"(["dt"])", default_fields); + Status not_implemented = + catalog->GetFormatTable(Identifier("db1", "catalog_partitions")).status(); + ASSERT_TRUE(not_implemented.IsNotImplemented()) << not_implemented.ToString(); +} + TEST_F(RestCatalogTest, NestedSchemaHighestFieldId) { ASSERT_OK_AND_ASSIGN(std::unique_ptr catalog, CreateRestCatalog()); ASSERT_OK(catalog->CreateDatabase("db1", {}, /*ignore_if_exists=*/false)); @@ -796,8 +1003,11 @@ TEST_F(RestCatalogTest, PartitionKeysRoundTrip) { arrow::field("f1", arrow::utf8(), /*nullable=*/false)}); struct ArrowSchema c_schema; ASSERT_TRUE(arrow::ExportSchema(*schema, &c_schema).ok()); + // The server's config sets a `table-default.bucket`, which would make this a bucketed append + // table - and one of those needs a bucket key it has not been given. ASSERT_OK(catalog->CreateTable(Identifier("db1", "pt"), &c_schema, - /*partition_keys=*/{"f1"}, /*primary_keys=*/{}, {}, + /*partition_keys=*/{"f1"}, /*primary_keys=*/{}, + {{"bucket", "-1"}}, /*ignore_if_exists=*/false)); // the partition keys survive both the create request and the load response conversion ASSERT_OK_AND_ASSIGN(std::shared_ptr
table, catalog->GetTable(Identifier("db1", "pt"))); diff --git a/src/paimon/rest/rest_messages.cpp b/src/paimon/rest/rest_messages.cpp index 324a8408..6606699d 100644 --- a/src/paimon/rest/rest_messages.cpp +++ b/src/paimon/rest/rest_messages.cpp @@ -345,6 +345,11 @@ void GetTableResponse::FromJson(const rapidjson::Value& obj) noexcept(false) { database_ = RapidJsonUtil::DeserializeKeyValue(obj, kFieldDatabase, std::string()); name_ = RapidJsonUtil::DeserializeKeyValue(obj, kFieldName, std::string()); path_ = RapidJsonUtil::DeserializeKeyValue(obj, kFieldPath); + // Everything built from this response reads and writes below the path, and an empty one + // bounds nothing. + if (path_.empty()) { + throw std::invalid_argument(std::string("member '") + kFieldPath + "' cannot be empty"); + } is_external_ = RapidJsonUtil::DeserializeKeyValue(obj, kFieldIsExternal, false); schema_id_ = RapidJsonUtil::DeserializeKeyValue(obj, kFieldSchemaId); schema_json_ = DeserializeRawJsonMember(obj, kFieldSchema); diff --git a/src/paimon/rest/rest_messages_test.cpp b/src/paimon/rest/rest_messages_test.cpp index 7a29b77c..3c76dfc1 100644 --- a/src/paimon/rest/rest_messages_test.cpp +++ b/src/paimon/rest/rest_messages_test.cpp @@ -211,6 +211,15 @@ TEST(RestMessagesTest, UnknownFieldsAreIgnored) { TEST(RestMessagesTest, MissingRequiredFieldsFail) { // "path", "schemaId" and "schema" are required ASSERT_NOK(GetTableResponse::FromJsonString(R"({"id": "42", "name": "t1"})").status()); + // A path that is there but empty is no path either: everything built from the response reads + // and writes below it, so a table whose paths were checked against an empty one would have no + // boundary at all. + ASSERT_NOK_WITH_MSG(GetTableResponse::FromJsonString( + R"({"id": "42", "name": "t1", "path": "", "schemaId": 3, "schema": )" + R"({"fields": [{"id": 0, "name": "f0", "type": "INT NOT NULL"}],)" + R"( "partitionKeys": [], "primaryKeys": [], "options": {}}})") + .status(), + "cannot be empty"); // a non-object identifier is rejected CreateTableRequest bad_identifier("", "", ""); ASSERT_NOK(RapidJsonUtil::FromJsonString(R"({"identifier": "not-an-object", "schema": {}})",