feat: size_of over all live DistArrays of a type in a World

[evaleev]

Summary

Adds a ground-truth memory-accounting facility: find every live DistArray of a given type in one or more Worlds by walking each World's WorldObject registry, instead of summing size_of over a set of handles.

Motivation: a DistArray is a shallow-copy handle, so summing size_of over handles double-counts arrays referenced from multiple places (e.g. the same tensor held both in an MPQC factory cache and in SeQuant's eval cache). Each array's tile storage is a single detail::DistributedStorage WorldObject, so walking the registry counts each array exactly once — usable as a cross-check against handle-based accounting.

API (TiledArray namespace, dist_array.h)

• size_of<DistArrayT, S>(World&) — walk world.get_object_ids(), recover each registered pointer as the common polymorphic base madness::WorldObjectBase, dynamic_cast to the DistributedStorage matching DistArrayT's tile type (non-matching objects skipped), sum DistributedStorage::local_size_in_bytes<S>().
• size_of_live_distarrays<S, DistArrayTs...>(worlds) — [world][type] matrix of the above.

DistributedStorage gains local_size_in_bytes<S>() (distributed_storage.h): sums TiledArray::size_of<S>(tile) over locally-owned, set futures — the same tile set size_of(DistArray) iterates.

Type-safety

The registry stores void* (registered as static_cast<Derived*>(this)), so a blind static_cast to the wrong type would be UB. The cast goes through madness::WorldObjectBase (the polymorphic base of WorldObject<Derived>) and then dynamic_cast, which yields nullptr for non-matching types. This is sound as long as WorldObjectBase sits at offset 0 of every registered WorldObject — true for the single-inheritance class X : public WorldObject<X> idiom. Verified across MADNESS, TiledArray, and MPQC that no WorldObject-derived class lists WorldObject as a non-primary base.

Caveats

• Counts only locally-owned tiles whose futures are set; excludes TiledRange/shape metadata and remote-tile caches. Call at a quiescent point (after a fence).
• S is the leading template arg of the variadic form (it has a default but precedes the pack): size_of_live_distarrays<MemorySpace::Host, ArrayA, ArrayB>(worlds).

Test plan

• array_suite/size_of_live_in_world (np=1): two distinct arrays + a shallow copy of one; asserts the walk equals the 2-array deduplicated tile total (not the 3-handle sum), a ToT-typed walk ignores regular-tensor arrays, and the variadic matrix form. Full array_suite green at np=1.
• np=2 — left to CI; the local macOS ASan+mpirun environment crashes all tests (verified an untouched pre-existing test crashes identically), so it can't validate np=2 locally.

[Copilot]

Pull request overview

Adds a “ground-truth” memory-accounting path for DistArray tile storage by walking a World’s WorldObject registry to find live detail::DistributedStorage<T> instances and summing locally-owned, already-set tiles (deduplicating shallow-copy handles). This complements existing handle-based size_of(DistArray) accounting, which can double-count shared storage.

Changes:

• Add detail::DistributedStorage::for_each_local_tile() to iterate locally-owned, set tile futures without communication.
• Add TiledArray::size_of(S, DistributedStorage), size_of_live_distarray_storage(World&), and a variadic size_of_live_distarrays_storage(worlds) “matrix” helper.
• Add a unit test validating deduplication across shallow copies and type-filtering behavior.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.

File	Description
tests/dist_array.cpp	Adds coverage for registry-walk storage accounting vs handle-based accounting (including shallow-copy deduplication and variadic API).
src/TiledArray/distributed_storage.h	Introduces a tile-iteration helper over locally-owned, already-set futures to support local-only accounting.
src/TiledArray/dist_array.h	Adds public APIs to compute live DistributedStorage tile-data sizes per World and across multiple types/worlds.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.