[Bug] Format table reads and deletes committer staging files inside partitions

[sundapeng]

Search before asking

• I searched in the issues and found nothing similar.

Paimon version

master (45cf5ffb0); every version that has Format Table support is affected.

Compute Engine

Any engine reading a Format Table through paimon-core; first seen on Trino over an object store
with a magic committer, but Spark and Flink go through the same code path.

Minimal reproduce step

1. Create a partitioned Format Table (parquet) and write a partition normally.
2. Leave a committer staging tree inside one partition — exactly what an S3A/JindoOSS magic
   committer or a FileOutputCommitter leaves behind while a job runs, or after it finishes with
   cleanup disabled:

<table>/dt=20260727/__magic_job-<id>/tasks/attempt_<id>/__base/part-00010-xxx.snappy.parquet  (0 bytes)
<table>/dt=20260727/__magic_job-<id>/tasks/attempt_<id>/__base/part-00010-xxx.snappy.parquet.pending

3. SELECT count(*) FROM <table> WHERE dt = '20260727'.

Query ... failed: ...__magic<uuid>/tasks/attempt_.../__base/part-00010-xxx.snappy.parquet
is not a Parquet file (length is too low: 0)

The whole query fails — not just that partition.

What doesn't meet your expectations?

Files under a hidden directory are not table data and must not be read, and must not be deleted by
someone else's INSERT OVERWRITE. Two places treat them as data, for the same reason: a
recursive listing is filtered by the leaf file name only.

1. Read — SplitEnumerator#createSplits
(paimon-core/src/main/java/org/apache/paimon/table/format/SplitEnumerator.java:112-118):

FileStatus[] files = fileIO.listFiles(path, true);          // recursive
for (FileStatus file : files) {
    if (FormatTableScan.isDataFileName(file.getPath().getName())) {   // leaf name only

FormatTableScan#isDataFileName
(paimon-core/src/main/java/org/apache/paimon/table/format/FormatTableScan.java:101-103) is
!name.startsWith(".") && !name.startsWith("_"). part-00010-xxx.snappy.parquet is a perfectly
ordinary name — it is only the __magic.../__base/ directories above it that say the file is
uncommitted. So the 0-byte placeholder is opened as parquet and the query dies.

2. Write — FormatTableCommit#deletePreviousDataFile
(paimon-core/src/main/java/org/apache/paimon/table/format/FormatTableCommit.java:270-284) does the
same listFiles(partitionPath, true) + isDataFileName(...) and then deletes. An
INSERT OVERWRITE therefore deletes the pending files of any writer that is still running in that
partition — data loss for that job, not just a failed read.

3. Cleanup — RenamingTwoPhaseOutputStream.TempFileCommitter#clean
(paimon-common/src/main/java/org/apache/paimon/fs/RenamingTwoPhaseOutputStream.java:138-140)
deletes tempPath.getParent(), i.e. the whole <partition>/_temporary directory, after committing
one file. _temporary is shared: two concurrent Paimon writers into the same partition delete each
other's pending files, and any Hadoop FileOutputCommitter job staging there loses its output.

Why nothing else hits this. Every other engine filters a recursive listing per path component.
Hive says it explicitly in HIDDEN_FILES_FULL_PATH_FILTER
(standalone-metastore/metastore-common/.../metastore/utils/FileUtils.java), which it applies to the
path made relative to the listing root in the S3A fast path of listStatusRecursively — the same
fs.listFiles(base, true) API Paimon uses:

This works with RemoteIterator which (potentially) produces all files recursively so looking for
hidden folders must look at whole path, not just the last part of it as would be appropriate
w/o recursive listing.

Hadoop MapReduce, Trino (containsHiddenPathPartAfterIndex(path, rootPrefixLength)) and Spark do the
same. #6522 fixed partition discovery in Paimon (PartitionPathUtils.searchPartSpecAndPaths
checks isHiddenFile at every level); the listing inside a partition never followed.

Anything else?

Two facts that make this more than a corner case:

• Leftover staging trees are a blessed steady state, not an anomaly.
  fs.s3a.committer.magic.cleanup.enabled=false (HADOOP-18568, Hadoop 3.4.2+) is the documented
  recommendation for large jobs, with the __magic tree removed by an object-store lifecycle rule
  instead; MagicS3GuardCommitter.cleanupStagingDirs() also swallows deletion failures. "It
  disappears when the job finishes" does not hold.
• Do not match on the literal __magic. Since HADOOP-18797 (Hadoop 3.4.0) the directory is
  named __magic_job-${jobId}. The generic _ / . prefix rule is the one to implement; it also
  covers _temporary, .hive-staging_*, _temporary_jindo, and matches JindoOSS's own default
  (fs.jfs.cache.oss.delete-marker.dirs = temporary,.staging,.hive-staging,__magic).

One correctness note for whoever fixes it: the hidden-component check must be relative to the
listed directory. Walking up to the filesystem root would make every table under a warehouse path
like oss://bucket/_warehouse/db/t read as empty — silently reading nothing is worse than the bug
being fixed.

Reproduced end to end on a real deployment: injecting a single 0-byte
__magic.../tasks/attempt_.../__base/part-*.snappy.parquet into one partition of an existing table
reproduces the error above verbatim, and filtering the staging tree out makes the same query return
its normal result with partition pruning unchanged.

Are you willing to submit a PR?

• I'm willing to submit a PR!

[sundapeng]

Closing this: the two PRs describe the problem directly, so the issue only adds indirection.

• [core][common] Fix format tables reading and deleting files staged by a committer #8861 — the recursive listing inside a partition (read, and INSERT OVERWRITE delete)
• 【wip】[common] Keep the shared _temporary directory while other writers stage there #8862 — clean() deleting the shared _temporary directory