Skip to content

Add __slots__ to Node and Path#4966

Draft
invisig0th wants to merge 1 commit into
masterfrom
node-path-slots
Draft

Add __slots__ to Node and Path#4966
invisig0th wants to merge 1 commit into
masterfrom
node-path-slots

Conversation

@invisig0th
Copy link
Copy Markdown
Contributor

Summary

Add __slots__ to synapse.lib.node.Node and synapse.lib.node.Path to remove the per-instance __dict__ and reduce memory for these high-cardinality runtime objects (a Node/Path pair is created for every node touched during Storm execution).

Details

  • Node.__slots__ includes:
    • __weakref__ -- Node instances are held in the snap livenodes WeakValueDictionary (synapse/lib/snap.py), so they must remain weak-referenceable.
    • isrunt -- set externally on the instance in synapse/lib/snap.py.
  • Path.__slots__ includes display, which is set externally in synapse/lib/storm.py.
  • Rewrote the coherency test in synapse/tests/test_lib_snap.py: it previously set a dynamic node._test attribute (which __slots__ forbids) to prove a re-fetched node was a different object. It now holds a weakref.ref to the original node and asserts it is collected after GC -- a more direct check that the node was released from the coherency cache.
  • Added a feat changelog entry.

Backward compatibility

__slots__ prevents setting arbitrary attributes on Node/Path instances. All in-tree assignments are accounted for; out-of-tree code that monkey-patches attributes onto these instances would need updating (noted in the changelog).

Testing

python -m pytest synapse/tests/test_lib_node.py synapse/tests/test_lib_snap.py    # 29 passed
python -m pytest synapse/tests/test_lib_storm.py synapse/tests/test_lib_ast.py     # 120 passed

Define __slots__ on synapse.lib.node.Node and Path to remove the
per-instance __dict__ and reduce memory for these high-cardinality
runtime objects. Node includes __weakref__ (it is held in the snap
livenodes WeakValueDictionary) and isrunt (set externally in snap).

Rewrite the snap coherency test to track the original node via a
weakref instead of setting a dynamic _test attribute.
@codecov
Copy link
Copy Markdown

codecov Bot commented May 30, 2026

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 97.76%. Comparing base (186ba33) to head (afed0f3).

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #4966      +/-   ##
==========================================
- Coverage   97.81%   97.76%   -0.05%     
==========================================
  Files         299      299              
  Lines       63433    63435       +2     
==========================================
- Hits        62047    62019      -28     
- Misses       1386     1416      +30     
Flag Coverage Δ
linux 97.70% <100.00%> (-0.05%) ⬇️
linux_replay 93.51% <100.00%> (+<0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@invisig0th
Copy link
Copy Markdown
Contributor Author

Performance: max Storm node throughput (before vs after __slots__)

Method. In-memory test Cortex (getTestCore), 100k test:int nodes pre-added, then timing test:int | spin — this forces a Node + Path per node through the full Storm pipeline and then discards them, i.e. the max-throughput path. 2 warmup + 6 measured trials; best = min wall time. Only synapse/lib/node.py was toggled between master (no slots) and this branch (slots). Python 3.14, single host.

Throughput

best mean
before (no slots) 34,843 nodes/s 34,591 nodes/s
after (slots) 34,885 nodes/s 34,480 nodes/s

No measurable throughput change. This path is dominated by LMDB reads, msgpack decode and async scheduling, not per-instance __dict__ allocation, so removing the dict doesn't move the pipeline rate.

Where the win actually is: memory

Average bytes per live Node, sys.getsizeof(obj) + sys.getsizeof(obj.__dict__), over 20k lifted nodes:

per Node
before (no slots) 200 B
after (slots) 136 B

-64 B/node (-32%). That delta is exactly the removed __dict__. At scale (large lifts / many simultaneously-live nodes) this is the real benefit; throughput is unchanged.

Caveat: in-memory test cortex on a single host under Python 3.14 (CI pins 3.11); absolute nodes/s will differ on real deployments, but the before/after comparison is apples-to-apples since only node.py changed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant