Skip to content

refactor(server): use papaya instead of DashMap for shards_table#2828

Closed
krishvishal wants to merge 2 commits intoapache:masterfrom
krishvishal:shard-papaya
Closed

refactor(server): use papaya instead of DashMap for shards_table#2828
krishvishal wants to merge 2 commits intoapache:masterfrom
krishvishal:shard-papaya

Conversation

@krishvishal
Copy link
Contributor

Which issue does this PR close?

Closes #2827

Rationale

shards_table has a read-heavy workload (queried on every message route, written only on partition create/delete/rebalance). papaya's lock-free design is purpose-built for this pattern.

What changed?

  • Replace DashMap with papaya::HashMap for the shards_table shared lookup table, which maps partition namespaces to owning shards.
  • All call sites updated to use papaya's pin-based guard API: pin() returns a HashMapRef with ergonomic get/insert/remove/iter methods. Only worry is that as long as pin() is not dropped the garbage collector of papaya won't clean memory. This maybe a cause for concern across async boundaries.

@codecov
Copy link

codecov bot commented Feb 27, 2026

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 67.59%. Comparing base (6f01169) to head (ff2dbf2).
⚠️ Report is 1 commits behind head on master.

Additional details and impacted files
@@             Coverage Diff              @@
##             master    #2828      +/-   ##
============================================
+ Coverage     67.58%   67.59%   +0.01%     
  Complexity      708      708              
============================================
  Files          1027     1027              
  Lines         83147    83150       +3     
  Branches      60004    60019      +15     
============================================
+ Hits          56195    56208      +13     
+ Misses        24612    24591      -21     
- Partials       2340     2351      +11     
Flag Coverage Δ
csharp 67.06% <ø> (-0.19%) ⬇️
go 6.84% <ø> (ø)
java 54.02% <ø> (ø)
node 92.26% <ø> (-0.07%) ⬇️
python 0.00% <ø> (ø)
rust 69.64% <100.00%> (+0.02%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
core/server/src/main.rs 66.34% <100.00%> (+0.21%) ⬆️
core/server/src/shard/builder.rs 100.00% <100.00%> (ø)
core/server/src/shard/communication.rs 83.47% <100.00%> (+0.13%) ⬆️
core/server/src/shard/mod.rs 90.25% <100.00%> (+7.69%) ⬆️
core/server/src/shard/system/streams.rs 96.52% <100.00%> (ø)
core/server/src/shard/system/topics.rs 90.05% <100.00%> (ø)

... and 9 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Copy link
Contributor

@numinnex numinnex left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's not do it for the IggyShard on the server, but rather only in the shard module that we will use in simulator, this way we can profile them both and decide.

@krishvishal
Copy link
Contributor Author

@numinnex understood. Closing this PR.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Replace DashMap with papaya for shards_table in server shard

4 participants