[core][common] Fix format tables reading and deleting files staged by a committer

[sundapeng]

Purpose

A committer stages its output inside the partition it writes, under a hidden directory, and the staged files carry ordinary data file names:

<table>/dt=20260727/part-00000.parquet                              committed
<table>/dt=20260727/__magic_job-<id>/tasks/attempt_<id>/part-00010  another job, still staging
<table>/dt=20260727/_temporary/attempt_<id>/part-00011              another job, still staging

A format table lists the partition recursively and then filters on the leaf file name only, so all three become splits. A zero-length placeholder then fails the query with ... is not a Parquet file (length is too low: 0), and INSERT OVERWRITE, driven by the same listing, deletes the other job's files. From the other side, TempFileCommitter#clean deletes the whole _temporary after every commit, and that directory is shared with everyone else writing there.

This PR skips hidden entries while walking instead of filtering names afterwards — what PartitionPathUtils#isHiddenFile already does for partition discovery (#6522) — and makes clean() drop only its own temporary file, removing the shared directory only when a non-recursive delete succeeds. The two halves are needed together: keeping a staging tree alive while the reader still treats it as data would put uncommitted rows into query results.

partitionLevels marks the levels that hold partition directories. They are descended whatever they are named, because a partition value may start with _ — the default partition name does, in the value-only layout — and INSERT OVERWRITE PARTITION (year = '2025') deletes under a prefix.

Brief change log

• FormatTableScan#listDataFiles(fileIO, root, partitionLevels): new; walks with listStatus and prunes hidden subtrees. Used by SplitEnumerator#createSplits and FormatTableCommit#deletePreviousDataFile.
• RenamingTwoPhaseOutputStream.TempFileCommitter#clean: delete this stream's temporary file; remove the shared directory only if a non-recursive delete succeeds.
• PartitionPathUtils#isHiddenName: extracted from isHiddenFile.

Tests

• FormatTableScanTest / FormatTableCommitTest: staged files produce no splits in either partition layout; a table under a _-prefixed path still reads; INSERT OVERWRITE leaves both staging trees intact yet still clears the default partition directory.
• RenamingTwoPhaseOutputStreamTest: a peer's pending file survives clean(); an uncontended commit leaves no _temporary behind.
• Every new case fails without the change; mvn -pl paimon-common,paimon-core test is green.

API and Format

• No storage format change; PartitionPathUtils gains one public helper (isHiddenName).

Documentation

• No documentation change needed