Add __slots__ to Node and Path#4966
Conversation
Define __slots__ on synapse.lib.node.Node and Path to remove the per-instance __dict__ and reduce memory for these high-cardinality runtime objects. Node includes __weakref__ (it is held in the snap livenodes WeakValueDictionary) and isrunt (set externally in snap). Rewrite the snap coherency test to track the original node via a weakref instead of setting a dynamic _test attribute.
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #4966 +/- ##
==========================================
- Coverage 97.81% 97.76% -0.05%
==========================================
Files 299 299
Lines 63433 63435 +2
==========================================
- Hits 62047 62019 -28
- Misses 1386 1416 +30
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
Performance: max Storm node throughput (before vs after
|
| best | mean | |
|---|---|---|
| before (no slots) | 34,843 nodes/s | 34,591 nodes/s |
| after (slots) | 34,885 nodes/s | 34,480 nodes/s |
No measurable throughput change. This path is dominated by LMDB reads, msgpack decode and async scheduling, not per-instance __dict__ allocation, so removing the dict doesn't move the pipeline rate.
Where the win actually is: memory
Average bytes per live Node, sys.getsizeof(obj) + sys.getsizeof(obj.__dict__), over 20k lifted nodes:
per Node |
|
|---|---|
| before (no slots) | 200 B |
| after (slots) | 136 B |
-64 B/node (-32%). That delta is exactly the removed __dict__. At scale (large lifts / many simultaneously-live nodes) this is the real benefit; throughput is unchanged.
Caveat: in-memory test cortex on a single host under Python 3.14 (CI pins 3.11); absolute nodes/s will differ on real deployments, but the before/after comparison is apples-to-apples since only node.py changed.
Summary
Add
__slots__tosynapse.lib.node.Nodeandsynapse.lib.node.Pathto remove the per-instance__dict__and reduce memory for these high-cardinality runtime objects (aNode/Pathpair is created for every node touched during Storm execution).Details
Node.__slots__includes:__weakref__--Nodeinstances are held in the snaplivenodesWeakValueDictionary(synapse/lib/snap.py), so they must remain weak-referenceable.isrunt-- set externally on the instance insynapse/lib/snap.py.Path.__slots__includesdisplay, which is set externally insynapse/lib/storm.py.synapse/tests/test_lib_snap.py: it previously set a dynamicnode._testattribute (which__slots__forbids) to prove a re-fetched node was a different object. It now holds aweakref.refto the original node and asserts it is collected after GC -- a more direct check that the node was released from the coherency cache.featchangelog entry.Backward compatibility
__slots__prevents setting arbitrary attributes onNode/Pathinstances. All in-tree assignments are accounted for; out-of-tree code that monkey-patches attributes onto these instances would need updating (noted in the changelog).Testing