Antalya 26.6: Backport of 98178 Iceberg compaction using optimize

[subkanthi]

ClickHouse#98178

Changelog category (leave one):

• New Feature

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

Add support for iceberg compaction of manifest files using optimize command.

Documentation entry for user-facing changes

Adds OPTIMIZE TABLE ... MANIFEST to trigger manifest-only compaction for Iceberg tables, backport of ClickHouse#98178
This PR covers rewriting manifest files. The default value is 30, but can be overwritten using iceberg_manifest_min_count_to_compact. The setting is the threshold: compaction runs only if the current snapshot's manifest list contains strictly more than this many manifest files. Setting this value to 0 means "always compact"

CI/CD Options

Exclude tests:

• Fast test
• Integration Tests
• Stateless tests
• Stateful tests
• Performance tests
• Aarch64 tests
• All with ASAN
• All with TSAN
• All with MSAN
• All with UBSAN
• All with Coverage
• All Regression
• Disable CI Cache

Regression jobs to run:

• Fast suites (mostly <1h)
• Aggregate Functions (2h)
• Alter (1.5h)
• Benchmark (30m)
• ClickHouse Keeper (1h)
• Iceberg (2h)
• LDAP (1h)
• OAuth (5m)
• Parquet (1.5h)
• RBAC (1.5h)
• SSL Server (1h)
• S3 (2h)
• S3 Export (2h)
• Swarms (30m)
• Tiered Storage (2h)

[github-actions]

Workflow [PR], commit [9200f13]

[subkanthi]

OPTIMIZE TABLE iceberg_trino_test.`default.optimize_manifest_fieldid` MANIFEST
SETTINGS
    allow_experimental_iceberg_compaction = 1,
    iceberg_manifest_min_count_to_compact = 2,
    allow_insert_into_iceberg = 1,
    write_full_path_in_iceberg_metadata = 1;

test_storage_iceberg_with_trino/test.py::test_optimize_manifest_trino_field_ids
[gw0] [100%] PASSED test_storage_iceberg_with_trino/test.py::test_optimize_manifest_trino_field_ids

• generated report log file: /root/Documents/ClickHouse/ci/tmp/pytest_parallel.jsonl -
  ======================== 8 passed in 340.20s (0:05:40) =========================
  Run command: [sudo iptables -D DOCKER-USER 1 ||:]
  ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
  Ok [Integration tests (amd_tsan, 1/6)]
  | Failures: 0/8
  ++++++++++++++++++++

[subkanthi]

Multiple inserts to create manifests

Ubuntu-2404-noble-amd64-base :) insert into ice.`flowers.sample` values(14, false, 14.0);

INSERT INTO ice.`flowers.sample` FORMAT Values

Query id: 3efe4235-c9fc-4801-8388-e8424d9052c0

Ok.

1 row in set. Elapsed: 1.471 sec. 

Ubuntu-2404-noble-amd64-base :) insert into ice.`flowers.sample` values(15, false, 15.0);

INSERT INTO ice.`flowers.sample` FORMAT Values

Query id: e5c02b18-8a7c-4988-b406-6c76ef9ce1e3

Ok.

1 row in set. Elapsed: 4.284 sec. 

Ubuntu-2404-noble-amd64-base :) insert into ice.`flowers.sample` values(16, false, 16.0);

INSERT INTO ice.`flowers.sample` FORMAT Values

Query id: 579f1ff8-0e8c-4090-8401-4f8ad65cb85c

Ok.

1 row in set. Elapsed: 1.161 sec. 

Ubuntu-2404-noble-amd64-base :) insert into ice.`flowers.sample` values(17, false, 17.0);

INSERT INTO ice.`flowers.sample` FORMAT Values

Query id: 3ab42808-b5b5-4e00-894a-7da86685436e

Ok.

1 row in set. Elapsed: 1.293 sec. 

Ubuntu-2404-noble-amd64-base :) insert into ice.`flowers.sample` values(18, false, 18.0);

INSERT INTO ice.`flowers.sample` FORMAT Values

Query id: 1f0f3745-2df9-46b8-a2f3-24e80eebdc4b

Ok.

1 row in set. Elapsed: 0.722 sec. 

BEFORE:

ice files flowers.sample
2026-08-20 00:15:36 [main/1725085] INFO o.a.i.r.auth.AuthManagers > Loading AuthManager implementation: org.apache.iceberg.rest.auth.NoopAuthManager
Snapshots: flowers.sample
└── Snapshot 8262575200555583623, schema 4: s3://bucket1/flowers/sample/metadata/snap-8262575200555583623-2-2fc15c6b-7e72-4ded-8f47-9381d60aa78f.avro
    ├── Manifest: s3://bucket1/flowers/sample/metadata/58901361-d4dd-402c-8990-905b76fc8422.avro
    │   └── Datafile: s3://bucket1/flowers/sample/data/data-f53c47ab-8d0a-4b79-8edc-fbf960bce414.parquet
    ├── Manifest: s3://bucket1/flowers/sample/metadata/c2200427-403f-493a-8737-e98f10c820f3.avro
    │   └── Datafile: s3://bucket1/flowers/sample/data/data-6c3bcbcd-9f3a-4d8e-a2aa-e89dba45933d.parquet
    ├── Manifest: s3://bucket1/flowers/sample/metadata/16e42c94-0f1b-4fcb-96a5-90b3aa890676.avro
    │   └── Datafile: s3://bucket1/flowers/sample/data/data-9eca5943-09c3-423c-ae48-5fe3b89aad21.parquet
    ├── Manifest: s3://bucket1/flowers/sample/metadata/6794aab6-3dc9-4945-8528-31ed4023bf2f.avro
    │   └── Datafile: s3://bucket1/flowers/sample/data/data-20d9a7a8-bab8-4fa5-a665-d6d0552e73b7.parquet
    ├── Manifest: s3://bucket1/flowers/sample/metadata/685e041f-c818-48dc-955e-bdbe747e7afa.avro
    │   └── Datafile: s3://bucket1/flowers/sample/data/data-6200eb02-c3cf-4481-a1d6-8a62b748d9b1.parquet
    ├── Manifest: s3://bucket1/flowers/sample/metadata/120a6ca5-5ef0-4766-9cc8-b7eb4a06e986.avro
    │   └── Datafile: s3://bucket1/flowers/sample/data/1783515818468-5385203953de32f1e4f96effd36253b8cd6971f796a62acaf5bb9d7f45dd4249.parquet
    ├── Manifest: s3://bucket1/flowers/sample/metadata/d8fe3175-908e-45eb-b6f8-04587c794da7.avro
    │   ├── Datafile: s3://bucket1/flowers/sample/data/data-c26f7933-272c-4d98-bafb-1103038d7132.parquet
    │   ├── Datafile: s3://bucket1/flowers/sample/data/data-62238d2a-14af-4f85-a0b1-acd7c5b98294.parquet
    │   └── Datafile: s3://bucket1/flowers/sample/data/data-fa9aaac5-1043-4075-8a89-3c32735882ac.parquet
    └── Manifest: s3://bucket1/flowers/sample/metadata/c89ca381-e73f-468b-aebd-62b08815d002.avro
        └── Datafile: (failed to read: Cannot read a delete manifest with a ManifestReader: GenericManifestFile{content=DELETES, path=s3://bucket1/flowers/sample/metadata/c89ca381-e73f-468b-aebd-62b08815d002.avro, length=2944, partition_spec_id=0, added_snapshot_id=1745025533079389269, added_data_files_count=1, added_rows_count=1, existing_data_files_count=0, existing_rows_count=0, deleted_data_files_count=0, deleted_rows_count=0, partitions=null, key_metadata=null, sequence_number=2, min_sequence_number=2, first_row_id=null})

Ubuntu-2404-noble-amd64-base :) OPTIMIZE TABLE ice.`flowers.sample` MANIFEST
SETTINGS
    allow_experimental_iceberg_compaction = 1,
    iceberg_manifest_min_count_to_compact = 1,
    allow_insert_into_iceberg = 1,
    write_full_path_in_iceberg_metadata = 1;

AFTER

 files flowers.sample
2026-08-20 00:18:22 [main/1725551] INFO o.a.i.r.auth.AuthManagers > Loading AuthManager implementation: org.apache.iceberg.rest.auth.NoopAuthManager
Snapshots: flowers.sample
└── Snapshot 6210131662371618023, schema 4: s3://bucket1/flowers/sample/metadata/snap-6210131662371618023-2-53278c2e-c4b1-4fe3-92e1-199dad3bb964.avro
    ├── Manifest: s3://bucket1/flowers/sample/metadata/9b93d3ca-18c4-4bec-a754-3ef9c4e367c7.avro
    │   └── Datafile: s3://bucket1/flowers/sample/data/1783515818468-5385203953de32f1e4f96effd36253b8cd6971f796a62acaf5bb9d7f45dd4249.parquet
    ├── Manifest: s3://bucket1/flowers/sample/metadata/7d6b19df-decc-45b1-878d-4633ca419368.avro
    │   ├── Datafile: s3://bucket1/flowers/sample/data/data-f53c47ab-8d0a-4b79-8edc-fbf960bce414.parquet
    │   ├── Datafile: s3://bucket1/flowers/sample/data/data-6c3bcbcd-9f3a-4d8e-a2aa-e89dba45933d.parquet
    │   ├── Datafile: s3://bucket1/flowers/sample/data/data-9eca5943-09c3-423c-ae48-5fe3b89aad21.parquet
    │   ├── Datafile: s3://bucket1/flowers/sample/data/data-20d9a7a8-bab8-4fa5-a665-d6d0552e73b7.parquet
    │   ├── Datafile: s3://bucket1/flowers/sample/data/data-6200eb02-c3cf-4481-a1d6-8a62b748d9b1.parquet
    │   ├── Datafile: s3://bucket1/flowers/sample/data/data-c26f7933-272c-4d98-bafb-1103038d7132.parquet
    │   ├── Datafile: s3://bucket1/flowers/sample/data/data-62238d2a-14af-4f85-a0b1-acd7c5b98294.parquet
    │   └── Datafile: s3://bucket1/flowers/sample/data/data-fa9aaac5-1043-4075-8a89-3c32735882ac.parquet
    └── Manifest: s3://bucket1/flowers/sample/metadata/c89ca381-e73f-468b-aebd-62b08815d002.avro
        └── Datafile: (failed to read: Cannot read a delete manifest with a ManifestReader: GenericManifestFile{content=DELETES, path=s3://bucket1/flowers/sample/metadata/c89ca381-e73f-468b-aebd-62b08815d002.avro, length=2944, partition_spec_id=0, added_snapshot_id=1745025533079389269, added_data_files_count=1, added_rows_count=1, existing_data_files_count=0, existing_rows_count=0, deleted_data_files_count=0, deleted_rows_count=0, partitions=null, key_metadata=null, sequence_number=2, min_sequence_number=2, first_row_id=null})

[subkanthi]

@blau-ai

[blau-ai]

CI triage for #2193

Verdict: 7 red checks — 0 caused by this PR's code. 6 are flaky/infra/pre-existing; 1 is the expected consequence of adding a new setting and is fixed in the clickhouse-regression repo (snapshot update), not here. None of the failing tests touch the Iceberg/OPTIMIZE code paths this PR changes.

Head SHA analyzed: 9200f13.

---

Not PR-caused (flaky / infra / pre-existing)

1. Stateless tests (amd_debug, parallel) — 3 sub-failures, all cleared by CI's own diagnosis:

• 00071_merge_tree_optimize_aio — reproducible only with randomized settings (68/68 pass without randomization; CI verdict: "Confirmed: the failure is caused by randomized settings"). It's a MergeTree AIO test, not the Iceberg OPTIMIZE ... MANIFEST path. Flaky.
• 02456_keeper_retries_during_insert — "All reruns passed… not reproducible"; transient UNKNOWN_STATUS_OF_INSERT (quorum timeout). Flaky.
• 00163_column_oriented_formats — "All reruns passed… not reproducible"; Timeout! Killing process group. Flaky (see CI: stateless tests time out due to network latency between our runners and the S3 test-data bucket #2227).

2. Stateless tests (amd_debug, distributed plan, s3 storage, parallel)

• 04510_s3_explicit_url_named_secret_mask — Timeout!, failed 0 out of 3 reruns. The "signal 20 (Stopped)" stack is the framework's SIGTSTP kill on timeout, not a crash. Matches the known S3-runner latency issue CI: stateless tests time out due to network latency between our runners and the S3 test-data bucket #2227. Flaky/infra.

3. SQLLogic test — 6 new failures vs 10,520 fixed. Every new failure is Code: 241 … (total) memory limit exceeded … RSS 25.85 GiB on random/select/slt_good_* SELECT queries — memory-pressure flakiness on the runner, unrelated to OPTIMIZE/Iceberg.

4. Regression release settings — 1 of the 2 failing scenarios:

• max_alter_threads — snapshot expects auto(16), server reports auto(8). This is auto(<#cpu>); the snapshot was recorded on a 16-core box, this runner has 8. Hardware-dependent, not PR-related.

5. Regression release tiered_storage_cas — every failure is MetadataStorageFactory: unknown metadata storage type: cas. (UNKNOWN_ELEMENT_IN_CONFIG). The cas metadata-storage feature isn't available in this build; the CAS regression jobs were only just added to the pipeline (7419377). Independent of this diff (the PR touches no disk/metadata-storage config). Infra/build-config.

6. Grype Scan (…-alpine) — 1 High: CVE-2026-14456 in the Alpine base image. Container base-image CVE, nothing to do with the C++ changes here.

7. PR — aggregate gate; red only because of the above.

---

PR-related, but no code fix needed here

Regression release settings — the other failing scenario is iceberg_manifest_min_count_to_compact, reported as a SnapshotNotFoundError (name not present in the stored snapshot; actual value default=30).

This is the new setting this PR adds (src/Core/Settings.cpp, UInt64, default 30). The default_values regression test compares every setting against a stored snapshot, and this new name isn't in it yet. The actual default (30) is correct — this is the normal "new setting → update the snapshot" bookkeeping, not a bug in the PR.

Concrete fix (in Altinity/clickhouse-regression, not this repo): add an entry for iceberg_manifest_min_count_to_compact (default 30) to
settings/tests/snapshots/default_values.py.default values>=26.6_antalya.snapshot, typically by re-recording the 26.6 settings snapshot. I can't push there — it's outside my write scope — so flagging for whoever maintains the regression snapshots.

---

Suggested next steps

• Stateless + SQLLogic: safe to re-run — all diagnosed non-reproducible / randomized-settings / memory-pressure.
• settings / tiered_storage_cas / Grype: environmental (snapshot bookkeeping, missing cas build feature, base-image CVE) — handle at the regression/pipeline level, not in this PR.
• Nothing in the PR's C++ diff needs changing for these failures.

One coverage note (not a failure): all Integration tests jobs — including this PR's own test_manifest_compaction.py — were skipped in this pipeline, so the new Iceberg-compaction integration tests didn't actually execute here. Worth a targeted integration run to confirm the feature before merge.

_{🤖 automated CI triage · evidence from praktika result_pr.json, regression report.html, and Grype results for 9200f13}