feat(overlay): gate overlay writes on an explicit dataset setting

[wjones127]

Stacked on #8016 — review the last commit only.

Overlays were permitted on any dataset, so there was nowhere to record that a table wants overlays before its first one exists. That's the fact a high-level writer (update, merge_insert) needs to consult when deciding whether to store an update as an overlay.

Where enablement lives

lance.overlays.enabled, a table config key — not feature flag 64. Two reasons:

• It survives legacy writers. A pre-overlay writer recomputes reader/writer flags from what it knows and would drop bit 64, silently disabling the feature. Manifest::new_from_previous clones config verbatim, so a config key round-trips through writers that have never heard of it.
• It doesn't lock out readers early. reader_feature_flags is a hard refusal gate. An enabled-but-overlay-free dataset is byte-for-byte readable by a pre-overlay reader, and setting bit 64 on it would exclude them for no correctness reason. This matters more once the default flips: enablement-as-flag would set bit 64 on every newly created table at creation.

Flag 64 keeps its existing meaning — overlays are present.

Resolution, and the coming default flip

1. An explicit true/false wins.
2. Otherwise, a dataset already carrying overlays counts as enabled. This is what keeps datasets written while overlays were experimental writable, with no migration.
3. Otherwise DEFAULT_OVERLAYS_ENABLED, currently false.

The plan is to flip step 3 to true in ~6 months so existing tables get the optimization, which is a one-line change plus deleting clause 2's rationale. Two consequences are baked in deliberately:

• WriteParams::enable_overlays is Option<bool>, not bool. rust/CLAUDE.md says to name booleans so false/Default::default() is the desired default, which makes enable_*: bool wrong once the default is on — and renaming a public field to disable_* later is a break.
• The docs tell readers that a dataset which must not accumulate overlays should set false explicitly rather than relying on absence, since absence is the thing designed to change.

Not in this PR, and I'd treat it as a prerequisite for the flip: a session-level default, so a deployment can opt out fleet-wide before upgrading rather than table-by-table after. Resolution order would become explicit config → session default → library default; overlays_enabled_with is the single place it slots into.

The disable guard

One invariant — "disabled implies no overlays" — covers both directions: disabling while overlays remain, and committing a DataOverlay while disabled. It lives in build_manifest rather than at the API boundary because build_manifest re-runs on conflict rebase. check_update_config_txn puts UpdateConfig vs DataOverlay in the compatible bucket, so a disable racing an overlay write gets rebased and replayed against the newer manifest, and the check fires. test_disable_racing_a_concurrent_overlay_is_rejected covers exactly that path.

One subtlety worth a look: enablement is resolved against the pre-commit fragment list (overlays_enabled_with(&manifest.config, was_overlaid)). Resolving against post-commit fragments would let a DataOverlay authorize itself — the overlays it's adding would satisfy clause 2.

API

• WriteParams::enable_overlays: Option<bool> — honored on Create and Overwrite; append leaves the setting alone.
• Dataset::overlays_enabled() / Dataset::set_overlays_enabled(bool).
• Dataset::remove_overlays() — compacts overlaid fragments and nothing else. This needed a new CompactionOptions::overlays_only: max_overlays_per_fragment: Some(0) alone still bins small neighbors together and materializes deletions, and the obvious workaround of zeroing target_rows_per_fragment is unsafe because it doubles as max_rows_per_file for the rewrite. test_without_overlays_only_the_size_trigger_still_applies pins the contrast.

Test churn

Every existing overlay test committed overlays against datasets that never enabled them, so ~80 now correctly fail without opting in. Fixed by setting enable_overlays: Some(true) in each module's base-dataset helper (fragment.rs overlay_read, optimize.rs, dataset_overlay_index_masking.rs) and adding overlay_enabled_manifest() for the one transaction.rs unit test. That churn is the feature working.

New tests: 8 resolution cases + parse_overlays_enabled in lance-table; 7 e2e in dataset_overlay_feature_flag.rs (default-off, enable-at-creation, enablement surviving flag clearing, disable rejected with overlays, remove-then-disable, the rebase race, unparsable value, and a dataset with overlays but no key staying writable); 3 in optimize.rs for overlays_only.

Testing

cargo fmt --all and cargo clippy --all --tests --benches -- -D warnings clean. Full cargo test -p lance --lib green (2619 passed) and cargo test -p lance-table --lib green (135). cargo check clean for the python and java crates.

🤖 Generated with Claude Code

[github-actions]

Important

This PR touches the Lance format specification.

Substantive changes to the format specification — the .proto definitions
and the spec docs under docs/src/format/ — require a PMC vote before merge.
Minor edits such as typo fixes, wording, or formatting are excluded; use your
judgment.

If this is a meaningful format change:

• Start a vote following the Lance community voting process.
  Format specification modifications need 3 binding +1 votes (excluding the
  proposer), held on GitHub Discussions, with a minimum voting period of 1 week.
• Once the vote passes, link the completed vote in this PR. It should not be
  merged until the vote is linked.

[wjones127]

Folded into #8016 rather than kept as a stack. GitHub auto-marked this "merged" because its head became an ancestor of its base once will/oss-overlay-ga was fast-forwarded — nothing was merged to main. All review should happen on #8016, which now carries both commits.