count() trusts a corrupted snapshot-summary total-records verbatim instead of the manifest row counts

[Selfeer]

Describe the situation

On an Iceberg table without any deletes, SELECT count() now returns the total-records value from the snapshot summary verbatim. If that value is wrong — a corrupted commit, or a writer that maintains the incremental totals incorrectly — count() returns the bogus number while a scan of the same snapshot returns the real rows. The previous build protected against exactly this: it computed the count by summing the per-data-file record_count fields from the manifest files (required, per-file values) and only logged a warning when the summary disagreed.

Because the summary totals are maintained incrementally (parent total plus this commit's delta), a single bad commit anywhere in the table history silently poisons the totals of every later snapshot — so the wrong count() does not heal until the table is rewritten.

Found by

Regression suite test (rerun from the iceberg suite directory):

python3 regression.py --local --minio-root-user admin --minio-root-password password --clickhouse <build url or path> --only "/iceberg/deletion vectors/count paths/trivial count optimization/manifest sum wins/*" -l manifest_sum.log

---

How to reproduce the behavior

Environment

• Version: 26.6.2.20000.altinityantalya
• Writer: Spark with Iceberg (e.g. tabulario/spark-iceberg), REST catalog, MinIO storage

Steps

1. In Spark, create a plain Iceberg table with 50 rows and no deletes:

CREATE TABLE demo.db.tbl (id BIGINT, data STRING)
USING iceberg
TBLPROPERTIES ('format-version' = '3');

INSERT INTO demo.db.tbl
SELECT id, concat('row-', CAST(id AS STRING)) FROM range(50);

2. Simulate a poisoned commit history: in the current metadata JSON file under the table's metadata/ directory, set "total-records" in every snapshot's "summary" to "999999" and upload the file back. (This stands in for any writer bug or corrupted commit that breaks the incremental totals; the data files and manifests are untouched and fully consistent.)

3. In ClickHouse, clear the metadata cache and compare the count fast path against a real scan:

SYSTEM DROP ICEBERG METADATA CACHE;

SELECT count()
FROM icebergS3('http://minio:9000/warehouse/db/tbl', '<key>', '<secret>')
SETTINGS optimize_trivial_count_query = 1, use_iceberg_metadata_files_cache = 0;

SELECT count()
FROM (SELECT * FROM icebergS3('http://minio:9000/warehouse/db/tbl', '<key>', '<secret>'))
SETTINGS use_iceberg_metadata_files_cache = 0;

---

Expected behavior

count() returns 50 — the sum of the manifest record_count fields, which describe the actual data files of the snapshot. This is what the previous build returned, along with a server-log warning that the snapshot summary was inconsistent with the table data.

---

Actual behavior

The corrupted summary value is returned as the query result:

SELECT count() FROM icebergS3(...)

┌─count()─┐
│  999999 │
└─────────┘

SELECT count() FROM (SELECT * FROM icebergS3(...))

┌─count()─┐
│      50 │
└─────────┘

---

Question

Which source should count() treat as ground truth for append-only tables — the snapshot-summary total-records (cheap: no manifest reads) or the sum of manifest record_count (correct even when the incremental totals are poisoned)? The previous build deliberately chose the manifest sum because poisoned summary totals had been observed in real tables. If trusting the summary is now the intended trade-off for performance, please confirm so the test expectations can be aligned — but note that this failure mode is silent, unbounded (any count value is possible), and permanent for the affected table history. A middle ground could be a setting that selects between the two, or validating the summary against the manifest list without opening the manifest files.

This issue arose after the changes in: #2183

[ianton-ru]

Must be fixed in #2183