feat: shard directory namespace manifest

[jackye1995]

Summary

• add manifest_shard_count support for directory namespace manifests
• shard __manifest into routed physical fragments while keeping the manifest schema unchanged
• extend the manifest benchmark harness and docs for sharded S3 runs

Benchmark

• c7i.48xlarge, S3 us-east-1, jack-devland-build, write-create-namespace
• 1M inline continuous: 0.339 -> 2.859 ops/s (8.43x)
• 1M inline c=10: 0.324 -> 3.861 ops/s (11.92x)
• 1M no-index continuous: 0.490 -> 2.918 ops/s (5.96x)

[codecov]

Codecov Report

❌ Patch coverage is 77.39381% with 314 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
rust/lance-namespace-impls/src/dir/manifest.rs	73.54%	261 Missing and 52 partials ⚠️
rust/lance-namespace-impls/src/dir.rs	99.51%	1 Missing ⚠️

📢 Thoughts on this report? Let us know!

[jackye1995]

Update: the branch now also rebuilds the inline scalar index section for sharded manifest rewrites. The detailed S3 benchmark above was collected before this follow-up, when sharded rewrites skipped inline index rebuilds, so those numbers should be treated as pre-index-build results. We should rerun the same S3 panel before using the benchmark as final merge evidence.

[jackye1995]

S3 benchmark: sharded manifest with shard-local inline index segments

This reruns the S3 manifest commit benchmark after the sharded manifest index fix:

• Sharded rewrites rebuild only the touched shard's object_id, object_type, and base_objects scalar index segments.
• Scalar index search now prunes logical index segments to the scanner's explicit fragment set, so a routed shard lookup does not search all same-name manifest index segments.

Setup:

• Commit: becf7bae8603ee2850e597e27ffb27826c3fd465
• EC2: c7i.48xlarge
• S3: us-east-1, bucket jack-devland-build
• Operation: write-create-namespace
• Harness: rust/lance-namespace-impls/examples/manifest_bench.rs via rust/lance-namespace-impls/benches/manifest_commit_sweep.sh
• Inline manifest indices enabled only
• Size panel: 1k, 10k, 100k, 1M
• Concurrency panel: continuous 100 commits, plus c=10 and c=100 for 30s
• Runs were sequential on one host to avoid S3 throttling cross-talk.
• Baseline was run once and reused for both sharded comparisons.

Run IDs:

• Baseline + 4 shards: 20260612T072732Z
• 10 shards: 20260612T075747Z

Continuous, 100 commits

rows	baseline ops/s, p50/p99	4 shards ops/s (speedup), p50/p99	10 shards ops/s (speedup), p50/p99
1,000	2.487, 388/596 ms	2.593 (1.04x), 371/664 ms	2.503 (1.01x), 373/586 ms
10,000	1.627, 598/895 ms	2.095 (1.29x), 451/726 ms	2.423 (1.49x), 392/619 ms
100,000	1.262, 780/1279 ms	1.629 (1.29x), 589/1010 ms	1.566 (1.24x), 635/874 ms
1,000,000	0.342, 2852/3874 ms	1.036 (3.03x), 950/1287 ms	1.163 (3.40x), 850/1237 ms

Concurrent, c=10 for 30s

rows	baseline ops/s, p50/p99, errors	4 shards ops/s (speedup), p50/p99, errors	10 shards ops/s (speedup), p50/p99, errors
1,000	3.193, 2062/10965 ms, 0	3.123 (0.98x), 1203/13248 ms, 2	3.214 (1.01x), 1716/12240 ms, 0
10,000	2.030, 3039/16894 ms, 1	2.635 (1.30x), 1872/14337 ms, 1	3.062 (1.51x), 1634/14408 ms, 0
100,000	1.590, 3818/19594 ms, 1	2.296 (1.44x), 2794/15882 ms, 0	2.329 (1.46x), 2272/17117 ms, 1
1,000,000	0.333, 15584/62843 ms, 0	1.208 (3.63x), 4148/24569 ms, 2	1.681 (5.05x), 3969/13969 ms, 0

Concurrent, c=100 for 30s

rows	baseline ops/s, p50/p99, errors	4 shards ops/s (speedup), p50/p99, errors	10 shards ops/s (speedup), p50/p99, errors
1,000	3.929, 11651/22366 ms, 75	3.959 (1.01x), 10858/18546 ms, 99	3.788 (0.96x), 10998/20674 ms, 99
10,000	2.459, 13788/24898 ms, 104	2.610 (1.06x), 12092/27563 ms, 94	3.522 (1.43x), 10934/21423 ms, 100
100,000	1.744, 13968/27047 ms, 116	2.400 (1.38x), 10436/23632 ms, 123	2.587 (1.48x), 10024/20827 ms, 119
1,000,000	0.341, 46994/74115 ms, 79	1.321 (3.87x), 14366/28651 ms, 110	1.850 (5.43x), 12089/25183 ms, 129

Takeaways

• The corrected inline-index sharded path restores the expected scaling win. At 1M rows, 10 shards improves continuous writes by 3.40x, c=10 by 5.05x, and c=100 by 5.43x.
• 10 shards is generally better than 4 shards once there is enough data per manifest. It wins clearly at 10k c=100 and 1M, while 1k remains dominated by fixed commit/index overhead.
• 4 shards is still a meaningful improvement for large manifests: 1M continuous is 3.03x, c=10 is 3.63x, and c=100 is 3.87x.
• High c=100 error counts are expected commit-chain contention/retry-budget misses, not data corruption. Sharding reduces per-commit work but does not remove single-version commit contention.