feat(dir-catalog): add reader/writer feature flags to __manifest

[jackye1995]

Summary

Adds forward-compatibility infrastructure to the directory-catalog __manifest dataset, mirroring the Lance table format's reader/writer feature flags but at the catalog-manifest layer.

• Persists two u64 bitmasks in the __manifest dataset's table_metadata (lance.namespace.manifest.reader_feature_flags / writer_feature_flags). Absent keys parse as 0, so every existing manifest stays universally compatible.
• A build refuses to read or write a manifest that sets a flag it does not understand, returning a clear "please upgrade" error instead of misreading it. Reader and writer checks are enforced centrally: in the manifest consistency wrapper, at catalog open, and on the copy-on-write mutation path.
• Also stops the directory catalog from silently degrading to directory listing when the manifest is incompatible — build() and the per-operation fallbacks propagate the incompatibility instead of masking it, so the check cannot be bypassed.

This is the mechanism only: no manifest feature is defined yet, so the known masks are 0 and nothing is ever set — zero behavior change today. It is the prerequisite so that a future __manifest format change (e.g. a schema migration) can be shipped safely: that change adds its bit to the known masks and stamps it on write, and from then on older clients refuse the new format instead of misreading it.

[codecov]

Codecov Report

❌ Patch coverage is 91.96429% with 18 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
rust/lance-namespace-impls/src/dir/manifest.rs	86.84%	11 Missing and 4 partials ⚠️
rust/lance-namespace-impls/src/dir.rs	50.00%	1 Missing and 1 partial ⚠️
...-namespace-impls/src/dir/manifest_feature_flags.rs	99.05%	0 Missing and 1 partial ⚠️

📢 Thoughts on this report? Let us know!

[LuQQiu]

@jackye1995 Claude review

  The direction and layering are right, and the error-classification implementation (matching the error code, not the message) is clean. But there is one P1 that must be
  fixed:

  P1: Writer-flag enforcement is leaky — most write paths never hit the check

  The two real mutation chokepoints are insert_into_manifest_with_metadata (manifest.rs:1099) and delete_from_manifest (manifest.rs:1154). Both clone the dataset out of get()
  — which now performs only ensure_readable — then commit directly via MergeInsertBuilder/DeleteBuilder and write back with set_latest. ensure_writable is never called on
  these paths.

  The PR adds an explicit ensure_manifest_writable() pre-check only to create_table. So these entry points sail straight through on a writer-flagged manifest:
  create_namespace, drop_namespace, drop_table, declare_table, register_table, deregister_table (plus the migration register_table helper). The covered paths are only the
  get_mut() ones (inline optimization, set_property), the copy-on-write rewrite path, and create_table.

  This defeats the exact guarantee the mechanism exists to provide: the day the first writer flag actually ships, an older build can still happily delete and insert rows in
  the new-format manifest. The PR body's claim that "reader and writer checks are enforced centrally" doesn't hold; and the new test only exercises create_table — the one
  patched path — so a green suite masks the hole. Also, the comment on ensure_manifest_writable saying "the eventual commit re-checks" isn't true today: the final
  MergeInsert/Delete commit performs no re-check.

  The fix is natural: move ensure_writable into the two chokepoint functions (right after acquiring manifest_mutation_lock and loading the fresh dataset via get()), which
  closes every write entry point at once. Keep create_table's early check — its value is refusing before table data is written, avoiding an orphaned dataset directory. Once
  fixed, the "re-checks" comment becomes accurate too.