SELECT count() returns wrong result on Iceberg v3 table after Spark compaction

[Selfeer]

Describe the situation

On an Iceberg v3 merge-on-read table written entirely by Spark (no crafted or hand-edited metadata), SELECT count() disagrees with SELECT * from the same table after Spark compacts the data files: the scan correctly returns 90 rows while count() returns 80.

Found by

Regression suite tests (rerun from the iceberg suite directory):

python3 regression.py --local --minio-root-user admin --minio-root-password password --clickhouse <build url or path> --only "/iceberg/deletion vectors/time travel/compaction/*" -l compaction.log
python3 regression.py --local --minio-root-user admin --minio-root-password password --clickhouse <build url or path> --only "/iceberg/deletion vectors/count paths/count from files cache/*" -l count_cache.log

---

How to reproduce the behavior

Environment

• Version: 26.6.2.20000.altinityantalya
• Writer: Spark with Iceberg (e.g. tabulario/spark-iceberg), REST catalog, MinIO storage

Steps

1. In Spark, create an Iceberg v3 merge-on-read table with 100 rows and delete 10 of them (the delete produces a deletion vector):

CREATE TABLE demo.db.tbl (id BIGINT, data STRING)
USING iceberg
TBLPROPERTIES (
    'format-version' = '3',
    'write.delete.mode' = 'merge-on-read',
    'write.update.mode' = 'merge-on-read',
    'write.merge.mode' = 'merge-on-read'
);

INSERT INTO demo.db.tbl
SELECT id, concat('row-', CAST(id AS STRING)) FROM range(100);

DELETE FROM demo.db.tbl WHERE id % 10 = 0;

2. In ClickHouse, verify the table reads correctly before compaction (both return 90):

SELECT count()
FROM icebergS3('http://minio:9000/warehouse/db/tbl', '<key>', '<secret>');

SELECT count()
FROM (SELECT * FROM icebergS3('http://minio:9000/warehouse/db/tbl', '<key>', '<secret>'));

3. In Spark, compact the table with the standard procedure:

CALL demo.system.rewrite_data_files(
    table => 'db.tbl',
    options => map('delete-file-threshold', '1')
);

4. In ClickHouse, read the table again:

SELECT count()
FROM icebergS3('http://minio:9000/warehouse/db/tbl', '<key>', '<secret>')
SETTINGS use_iceberg_metadata_files_cache = 0;

SELECT count()
FROM (SELECT * FROM icebergS3('http://minio:9000/warehouse/db/tbl', '<key>', '<secret>'))
SETTINGS use_iceberg_metadata_files_cache = 0;

---

Expected behavior

Both queries return the same number of rows — 90. Compaction rewrites the data files with the deletion vector already applied, so the logical content of the table is unchanged.

---

Actual behavior

The full scan is correct, the count() fast path is not:

SELECT count() FROM icebergS3(...)

┌─count()─┐
│      80 │
└─────────┘

SELECT count() FROM (SELECT * FROM icebergS3(...))

┌─count()─┐
│      90 │
└─────────┘

SELECT * also returns the correct 90 rows with the correct ids.

---

Observations

The wrong value is exactly 90 − 10. After compaction, the snapshot summary of the rewrite_data_files snapshot describes 90 total records but still carries total-position-deletes = 10, even though those deletes were already applied during the rewrite and no live delete files remain. The count() result matches "summary total records minus summary total position deletes", which suggests the count is answered from snapshot summary statistics and double-counts deletes that the compaction already materialized.

Snapshot-summary totals are incremental statistics maintained by writers; after a rewrite they do not necessarily describe the live snapshot content, so subtracting them cannot be relied on for a correct count.

Question

Should the metadata count() fast path be applied at all when the snapshot summary reports a non-zero total-position-deletes? Falling back to the manifest/scan path in that case would make count() agree with the scan.

Related to: #2183

[ianton-ru]

Must be fixed in #2183