feat(format): promote data overlay files to GA with an explicit enable setting

[wjones127]

Two commits, best reviewed in order.

1. Remove the unstable gate

Data overlay files were gated behind a build-profile check plus an escape hatch: release builds treated feature flag 64 as unknown and refused any dataset carrying an overlay, unless LANCE_ENABLE_UNSTABLE_DATA_OVERLAY_FILES was set.

• Drop data_overlay_files_enabled() / supported_flags_when() and the env var. can_read_dataset/can_write_dataset check a plain SUPPORTED_FLAGS constant again.
• Rename FLAG_UNSTABLE_DATA_OVERLAY_FILES to FLAG_DATA_OVERLAY_FILES (value unchanged at 64).
• Drop the "Experimental / not yet supported in any library" warnings from the format docs.

The flag lifecycle you asked about already worked. apply_feature_flags recomputes the flag from the manifest's fragments on every commit, so it is set by the commit that attaches the first overlay and cleared by the commit that removes the last one. No production change was needed; it was just untested, so this pins it: a unit test for set-then-clear, plus 4 e2e tests covering a DataOverlay commit setting it, deleting the overlaid fragments clearing it, compaction clearing it, and it staying set while any other fragment is overlaid. The clearing cases reopen the dataset so they assert against the persisted manifest.

2. Gate overlay writes on an explicit setting

Overlays were permitted on any dataset, so there was nowhere to record that a table wants overlays before its first one exists — which is what a high-level writer (update, merge_insert) needs to consult when deciding whether to store an update as an overlay.

Where enablement lives

lance.overlays.enabled, a table config key — not feature flag 64:

• It survives legacy writers. A pre-overlay writer recomputes reader/writer flags from what it knows and would drop bit 64, silently disabling the feature. Manifest::new_from_previous clones config verbatim, so a config key round-trips through writers that have never heard of it.
• It doesn't lock out readers early. reader_feature_flags is a hard refusal gate. An enabled-but-overlay-free dataset is byte-for-byte readable by a pre-overlay reader, and setting bit 64 on it would exclude them for no correctness reason. This matters more once the default flips: enablement-as-flag would set bit 64 on every newly created table.

Flag 64 keeps its existing meaning — overlays are present.

Resolution, and the coming default flip

1. An explicit true/false wins.
2. Otherwise, a dataset already carrying overlays counts as enabled. This keeps datasets written while overlays were experimental writable, with no migration.
3. Otherwise DEFAULT_OVERLAYS_ENABLED, currently false.

The plan is to flip step 3 to true in ~6 months so existing tables get the optimization — a one-line change. Two consequences are baked in deliberately:

• WriteParams::enable_overlays is Option<bool>, not bool. rust/CLAUDE.md says to name booleans so false/Default::default() is the desired default, which makes enable_*: bool wrong once the default is on, and renaming a public field to disable_* later is a break.
• The docs tell readers that a dataset which must not accumulate overlays should set false explicitly rather than relying on absence, since absence is the thing designed to change.

Not in this PR, and I'd treat it as a prerequisite for the flip: a session-level default, so a deployment can opt out fleet-wide before upgrading rather than table-by-table after. Resolution order would become explicit config → session default → library default; overlays_enabled_with is the single place it slots into.

The disable guard

One invariant — "disabled implies no overlays" — covers both directions: disabling while overlays remain, and committing a DataOverlay while disabled. It lives in build_manifest rather than at the API boundary because build_manifest re-runs on conflict rebase. check_update_config_txn puts UpdateConfig vs DataOverlay in the compatible bucket, so a disable racing an overlay write gets rebased and replayed against the newer manifest, and the check fires. test_disable_racing_a_concurrent_overlay_is_rejected covers that path.

One subtlety worth a look: enablement resolves against the pre-commit fragment list. Resolving against post-commit fragments would let a DataOverlay authorize itself, since the overlays it is adding would satisfy clause 2.

API

• WriteParams::enable_overlays: Option<bool> — honored on Create and Overwrite; append leaves the setting alone.
• Dataset::overlays_enabled() / Dataset::set_overlays_enabled(bool).
• Dataset::remove_overlays() — compacts overlaid fragments and nothing else. This needed a new CompactionOptions::overlays_only: max_overlays_per_fragment: Some(0) alone still bins small neighbors together and materializes deletions, and the obvious workaround of zeroing target_rows_per_fragment is unsafe because it doubles as max_rows_per_file for the rewrite.

Notes for review

• Public API break. FLAG_UNSTABLE_DATA_OVERLAY_FILES and ENABLE_UNSTABLE_DATA_OVERLAY_FILES_ENV were pub in lance-table and are removed rather than deprecated. Keeping "UNSTABLE" in the name after GA seemed worse than the break, and no released build could actually use the feature. Happy to add deprecated aliases.
• Test churn is large and intentional. Every existing overlay test committed overlays against datasets that never enabled them, so ~80 correctly started failing. Fixed at each module's base-dataset helper rather than per-test.
• The docs no longer name a release version as the first to support overlays (the removed TODO asked for one). Worth filling in at release time.
• Two doc stubs were stale in a way that contradicted GA: "Writer support" claimed single-file sparse overlays were unwritable (FileWriter::write_column advances one field at a time, so they are), and "Scheduling compaction" claimed compaction was unimplemented (max_overlays_per_fragment exists). Corrected both.

Testing

cargo fmt --all and cargo clippy --all --tests --benches -- -D warnings clean. Full cargo test -p lance --lib green (2619) and cargo test -p lance-table --lib green (135). cargo check clean for the python and java crates. Also ran the lance-table flag tests under --release to confirm the unstable gate is gone with debug_assertions off — they would have failed before.

🤖 Generated with Claude Code

[github-actions]

Important

This PR touches the Lance format specification.

Substantive changes to the format specification — the .proto definitions
and the spec docs under docs/src/format/ — require a PMC vote before merge.
Minor edits such as typo fixes, wording, or formatting are excluded; use your
judgment.

If this is a meaningful format change:

• Start a vote following the Lance community voting process.
  Format specification modifications need 3 binding +1 votes (excluding the
  proposer), held on GitHub Discussions, with a minimum voting period of 1 week.
• Once the vote passes, link the completed vote in this PR. It should not be
  merged until the vote is linked.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!