|
| 1 | +.. Licensed to the Apache Software Foundation (ASF) under one |
| 2 | +.. or more contributor license agreements. See the NOTICE file |
| 3 | +.. distributed with this work for additional information |
| 4 | +.. regarding copyright ownership. The ASF licenses this file |
| 5 | +.. to you under the Apache License, Version 2.0 (the |
| 6 | +.. "License"); you may not use this file except in compliance |
| 7 | +.. with the License. You may obtain a copy of the License at |
| 8 | +
|
| 9 | +.. http://www.apache.org/licenses/LICENSE-2.0 |
| 10 | +
|
| 11 | +.. Unless required by applicable law or agreed to in writing, |
| 12 | +.. software distributed under the License is distributed on an |
| 13 | +.. "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY |
| 14 | +.. KIND, either express or implied. See the License for the |
| 15 | +.. specific language governing permissions and limitations |
| 16 | +.. under the License. |
| 17 | +
|
| 18 | +.. Ported from the Paimon documentation: |
| 19 | +.. https://github.com/apache/paimon/blob/master/docs/docs/concepts/rest/tables.mdx |
| 20 | +
|
| 21 | +.. default-domain:: cpp |
| 22 | +.. highlight:: cpp |
| 23 | + |
| 24 | +Format Table |
| 25 | +============ |
| 26 | +A format table is a directory that holds multiple files of the same format. It carries no |
| 27 | +snapshots and no manifests: the files in the directory are the table, so reading it lists |
| 28 | +directories and writing to it adds files. A table is a format table when its ``type`` option is |
| 29 | +``format-table``; ``file.format`` then names the format of every file in it, which here is |
| 30 | +``parquet`` or ``orc``. |
| 31 | + |
| 32 | +A partitioned format table uses the standard Hive directory layout, and its partitions are |
| 33 | +discovered from that layout rather than from metadata. By default a partition directory is named |
| 34 | +``key=value``; setting ``format-table.partition-path-only-value`` names it by the value alone. |
| 35 | + |
| 36 | +Because a directory of plain files records no row identity, a format table only accepts inserts, |
| 37 | +and reads return the table's own columns with no ``_VALUE_KIND`` field. |
| 38 | + |
| 39 | +Reading and writing |
| 40 | +------------------- |
| 41 | +A format table is not served through :cpp:func:`Catalog::GetTable`, which describes a managed |
| 42 | +table; use :cpp:func:`Catalog::GetFormatTable` instead. It is then read and written through |
| 43 | +``FormatTableScan``, ``FormatTableRead``, ``FormatTableWrite`` and ``FormatTableCommit`` rather |
| 44 | +than through the managed table path. |
| 45 | + |
| 46 | +A write is two-phase, since a directory has no metadata to switch atomically: written files are |
| 47 | +staged under hidden names that a scan skips, and only the commit renames them into place. |
| 48 | + |
| 49 | +A ``FormatTableWrite`` and a ``FormatTableCommit`` are each driven by one thread, but separate |
| 50 | +ones may fill and add to a table at once: each write stages its files under a uuid of its own, and |
| 51 | +each commit publishes only the files its own messages name. Two *overwriting* commits over the |
| 52 | +same directory race, since an overwrite clears what is committed there before publishing anything. |
| 53 | +A ``FormatTableScan`` may be shared, since planning leaves it as it was. |
| 54 | + |
| 55 | +``TableRead::CreateCountReader()`` is not implemented for a format table, so counting its rows |
| 56 | +means reading them. That is a gap here rather than something the layout forces: ``parquet`` and |
| 57 | +``orc`` both record a row count in their own footer. |
| 58 | + |
| 59 | +A writer starts a new file once the one it is filling reaches ``target-file-row-num`` rows or |
| 60 | +``target-file-size`` bytes. Both are checked between batches rather than between rows, because a |
| 61 | +batch is the unit this API writes in, so a file may pass either target by up to one batch. Java |
| 62 | +checks the row count on every row and the size every thousand rows, and its files therefore sit |
| 63 | +closer to the target. |
| 64 | + |
| 65 | +Aborting a write |
| 66 | +---------------- |
| 67 | +``FormatTableWrite::Abort()`` removes the files the write staged. It is the one call still allowed |
| 68 | +after ``PrepareCommit()``, so a commit that is prepared and then abandoned can still be cleaned up. |
| 69 | + |
| 70 | +Path containment is checked on the path text, which stops a ``..`` from leaving the table but not |
| 71 | +a symbolic link pointing out of it - the same as Java's own local file system behaviour. |
| 72 | + |
| 73 | +``FormatTableCommit::Abort()`` does the same for the messages a commit was given. **Neither undoes |
| 74 | +a commit that succeeded**: once a file has been renamed into place it is no longer staged, and |
| 75 | +nothing here will take it back. Both are best effort and never fail, so a warning in the log is the |
| 76 | +only signal that a file could not be removed. |
| 77 | + |
| 78 | +Give ``FormatTableCommit`` only the messages this job's own writers produced. A message names a |
| 79 | +staged file by path, and a commit can tell that the path belongs to this table, sits in the |
| 80 | +partition the message declares, and is staged rather than already published - not whose staged file |
| 81 | +it is. A well-formed message from somewhere else is published, or discarded by ``Abort()``, like |
| 82 | +any other. |
| 83 | + |
| 84 | +Relationship to Java Paimon |
| 85 | +--------------------------- |
| 86 | +Java serves format tables from a Hive or REST catalog, which holds the schema. This implementation |
| 87 | +also serves them from a file system catalog, which keeps the schema under the table directory - an |
| 88 | +extension Java does not have. Only for such a table are the ``schema`` and ``branch`` directories |
| 89 | +below the location treated as metadata rather than as data. |
| 90 | + |
| 91 | +A file system catalog keeps a table's schema in ``schema`` and its branches in ``branch`` below |
| 92 | +the table location, so under ``format-table.partition-path-only-value`` the first partition value |
| 93 | +may not be ``schema`` or ``branch``: the directory a write would use is the one holding the |
| 94 | +table's own metadata. Such a write is refused, as is an overwrite naming that partition - which |
| 95 | +would otherwise delete the schema. A table served from a REST or Hive catalog keeps its schema |
| 96 | +elsewhere, so there these are ordinary partition values and are read and written like any other. |
| 97 | + |
| 98 | +Under that same layout a partition value may not start with ``_`` or ``.`` either, whichever |
| 99 | +catalog serves the table: the value is the whole directory name, and a scan skips every hidden |
| 100 | +name. Java writes such a directory and then cannot read it back; here the write is refused |
| 101 | +instead. The one exception is the value standing for a null partition, ``partition.default-name``, |
| 102 | +which the scan reads at a partition level by design. Under the ``key=value`` layout the question |
| 103 | +does not arise, since the key in front of the value keeps the directory name visible. |
| 104 | + |
| 105 | +Two smaller differences come from this library's own conventions: |
| 106 | + |
| 107 | +* a write takes one partition per batch: the batch declares it through |
| 108 | + ``RecordBatch::SetPartition()``, every row is checked against that declaration, and a batch |
| 109 | + mixing partitions is refused. Java routes row by row, so one write call there may land in any |
| 110 | + number of partitions; |
| 111 | +* a projection that names the same column twice is rejected when the read is built. Java reads |
| 112 | + such a column once per entry. |
| 113 | + |
| 114 | +Current limits |
| 115 | +-------------- |
| 116 | +Compared with Java Paimon, this implementation does not yet support: |
| 117 | + |
| 118 | +* the ``csv``, ``json``, ``text`` and ``mosaic`` file formats, leaving ``parquet`` and ``orc``. |
| 119 | + All four are line-delimited text in Java, which shares one line-reading layer between them; |
| 120 | + this library has no text file format at all, so the first of them to be added has to bring that |
| 121 | + layer with it; |
| 122 | +* cutting one large data file into byte ranges so that several readers share it. Java does this |
| 123 | + only for its line-delimited text formats, which are the ones missing here; ``parquet`` and |
| 124 | + ``orc`` each record where their own row groups and stripes begin, and a reader handed a byte |
| 125 | + range of one would have to find that out for itself; |
| 126 | +* ``metastore.partitioned-table``, which moves partition visibility into the catalog, and the |
| 127 | + Hive partition sync that goes with it; |
| 128 | +* partition filters beyond equality on partition values, where Java accepts a full predicate. |
| 129 | + Partition discovery here also lists one directory level at a time and applies the filter to each |
| 130 | + name, while Java turns a leading run of equality constraints into a path and starts listing |
| 131 | + below it; a table with many partitions therefore costs more listings here than in Java; |
| 132 | +* ``scan.ignore-corrupt-file`` and ``scan.ignore-lost-file``, which are not implemented: a |
| 133 | + corrupt or missing data file fails the read rather than being skipped; |
| 134 | +* ``partition.legacy-name``, which changes how a partition value is rendered into its directory |
| 135 | + name; |
| 136 | +* ``format-table.commit-hive-sync-url``, which registers committed partitions with a Hive |
| 137 | + metastore; |
| 138 | +* column default values. Java replaces a null in a column whose schema field declares a default |
| 139 | + with that default as it writes; here the null is written as it came; |
| 140 | +* a table every one of whose columns is a partition column. Java projects the partition columns |
| 141 | + out of what it writes, leaving files that carry nothing but a row count; here such a schema is |
| 142 | + refused when the table is created and when it is opened; |
| 143 | +* ``TIMESTAMP``, ``DECIMAL``, ``FLOAT`` and ``DOUBLE`` partition columns, which Java allows. This |
| 144 | + is a restriction of the whole library rather than of format tables, and a table Java created |
| 145 | + with such a partition column fails to open here rather than at the first read. |
| 146 | + |
| 147 | +``data-file.path-directory`` has no effect here, and none in Java either: Java's format table |
| 148 | +writer builds its paths from the table root rather than from that directory. |
| 149 | +``format-table.implementation`` is honoured by the engines rather than by the table - in Java |
| 150 | +Spark it selects between Paimon's own implementation and the engine's ``FileTable`` - so it has |
| 151 | +no meaning inside this library. |
| 152 | + |
| 153 | +Validation |
| 154 | +---------- |
| 155 | +A table Java can serve and this library cannot is refused at creation rather than accepted and |
| 156 | +then found unopenable, whichever catalog it is created through. It can still reach a catalog |
| 157 | +another way - written by Java, or by an older client - so the same checks run again when the |
| 158 | +table is opened. |
| 159 | + |
| 160 | +Whitespace in a partition value is judged by ASCII rules here, while Java uses |
| 161 | +``Character.isWhitespace``; a value made only of non-ASCII whitespace therefore lands in a |
| 162 | +partition of its own rather than in the default one. |
0 commit comments