Skip to content

perf: stamp issuer write coalescing#5392

Open
martinconic wants to merge 10 commits intomasterfrom
stamperstore-high-disk
Open

perf: stamp issuer write coalescing#5392
martinconic wants to merge 10 commits intomasterfrom
stamperstore-high-disk

Conversation

@martinconic
Copy link
Copy Markdown
Contributor

@martinconic martinconic commented Mar 9, 2026

Checklist

  • I have read the coding guide.
  • My change requires a documentation update, and I have done it.
  • I have added tests to cover my changes.
  • I have filled out the description and linked the related issues.

Description

Resolves #4884

This PR addresses extreme LevelDB I/O bottlenecks and CPU serialization overhead encountered during massive high-frequency chunk uploads.

The Problem

Under a load of hundreds of thousands of concurrent chunks, the node suffered from performance degradation across two distinct vectors:

  1. LevelDB Put Lock Saturation: Writing a stamp item for every chunk requires locking LevelDB. Under 1M concurrent pushsyncs, this sequentially locked the disk queue, choking disk writes up to >100MB/s and stalling the networking routines for >140 seconds cumulatively on CPU profiles.
  2. JSON Deserialization Overhead: Updating peer and surplus balances required fetching extensive math objects from the statestore. The unoptimized branch utilized JSON unmarshal to decode these values from ASCII text. Under heavy load, the CPU burned >20 seconds of pure flat execution time just parsing strings.

The Solution

  1. StamperStoreCache (Debounced Async Writes)
    • Wrapped the Stamper Store in a highly concurrent memory cache.
    • Network routines now instantly write payloads to a thread-safe map in RAM.
    • A background flush loop sweeps the map every 5 seconds, writing thousands of items into LevelDB in a single, atomic batch. Disk churn drops massively, and the master process is completely freed from I/O wait times.
  2. Binary BigInt Encoding
    • Implemented binary marshaling natively onto the central BigInt struct.
    • Refactored all accounting database getters/setters to utilize raw binary memory allocations instead of strings.
    • Included a silent fallback mechanism: the binary unmarshaler seamlessly parses older JSON-encoded database values so upgrades require no explicit database migration.

@martinconic martinconic changed the title feat: stamperstore high disk feat: stamperstore high disk usage Mar 11, 2026
@martinconic martinconic marked this pull request as ready for review March 12, 2026 13:26
@martinconic martinconic self-assigned this Mar 13, 2026
@acud
Copy link
Copy Markdown
Contributor

acud commented Mar 30, 2026

@martinconic thanks for this. I have some comments:

  1. this PR states it is about stamperstore disk usage, but modifies also the accounting package. those are different problems and it would be good to be a bit more meticulous about the scope of PRs
  2. there is no need to reinvent serialization for the big.Int package. it already supports multiple formats out of the box, please see the standard library documentation
  3. there is a shutdown logic added to the stamp persistence store. i am not sure whether this is needed on a package level or if we could have this on the application level. if we have components implementing this themselves we already are starting to duplicate logic. it is something that the application should provide. see the .DIRTY file as it touches this topic too. perhaps something similar is needed (or reusing of the same thing).
  4. bucket recovery logs in Debug (why not Error?) and swallows the error, the actual meaning of this is the future uploads of the user will override previously uploaded data. this is pretty severe.
  5. persisting the stamp buckets (in the loop with the ticket) emits warning, not error
  6. closing of the postage service reads off the channel it is closing, which should return immediately, not sure why it is needed

@acud acud changed the title feat: stamperstore high disk usage feat: stamp issuer write coalescing Apr 1, 2026
@acud acud changed the title feat: stamp issuer write coalescing perf: stamp issuer write coalescing Apr 1, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Stamperstore ldb writes produce high disk write

2 participants