feat: tiered data storage #7
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
enigbe
force-pushed
the
2025-10-tiered-data-storage
branch
9 times, most recently
from
29f47f3 to
264aa7f
Compare
enigbe
force-pushed
the
2025-10-tiered-data-storage
branch
3 times, most recently
from
a30cbfb to
1e7bdbc
Compare
…lices LDK gives us the actual funding output so we no longer need to create a dummy one with fake pubkeys
We insert a channel's funding utxo into our wallet so we can later calculate the fees for the transaction, otherwise our wallet would have incomplete information. We do it before the splice as we only really need this information for splices and not for all channels.
Exposes the funding_redeem_script that LDK already exposes
…wallet Insert channel funding outputs into Wallet
Refactor the unified_qr.rs module into unified.rs to provide a single API for sending payments to BIP 21/321 URIs and BIP 353 HRNs. This change simplifies the user interface by leveraging the bitcoin-payment-instructions library for parsing. Key changes: - Rename UnifiedQrPayment to UnifiedPayment. - Rename QRPaymentResult to UnifiedPaymentResult. - Update the send method to support both URIs and HRNs. - Update integration tests to match the new unified flow.
…tcoin-payment-instructions Refactor unified_qr.rs to use bitcoin-payment-instructions
Rather than using `KVStoreSync` we now use the async `KVStore` implementation for most `read_X` util methods used during node building. This is a first step towards making node building/startup entirely async eventually.
Previously, we would read entries of our payment store sequentially. This is more or less fine when we read from a local store, but when we read from a remote (e.g., VSS) store, all the latency could result in considerable slowdown during startup. Here, we opt to read store entries in batches.
Previously, we consistently handed around `Arc` references for most objects to avoid unnecessary refactoring work. This approach however introduced a bunch of unnecessary allocations through `Arc::clone`. Here we opt to rather use plain references in a bunch of places, reducing the usage of `Arc`s.
Add integration test that verifies 200 payments are correctly persisted and retrievable via `list_payments` after restarting a node. Co-Authored-By: Claude AI
…-payment-reads Parallelize `read_payments`
.. we bump to the most recent `rust-lightning` commit and fix some minor test code changes.
.. as this is now done by the background processor.
Bump the LDK dependency to current `main`
LDK 0.2 added a method to load `ChannelMonitor`s on startup without resilvering them, avoiding the startup latency of persistence for each `ChannelMonitor`. Here we start using it.
…silver Avoid resilvering `ChannelMonitor`s on startup
`LiquiditySource` takes a reference to our `PeerManager` but the `PeerManager` holds an indirect reference to the `LiquiditySource`. As a result, after our `Node` instance is `stop`ped and the `Node` `drop`ped, much of the node's memory will stick around, including the `NetworkGraph`. Here we fix this issue by using `Weak` pointers, though note that there is another issue caused by LDK's gossip validation API.
…tribution-api Bump LDK dependency for minor splicing API changes
In the last few days there was incompatibility of `cargo-semver-checks` with the new stable Rust 1.93.0. While this should fixed by today's release of `cargo-semver-checks`, we take the opportunity to drop an unnecessary install step from the CI workflow, as the action will bring their own Rust version if not configured otherwise.
…new-deref-api Update to new `rust-lightning` `UtxoSource` API
…ed-out-patch-section Add commented-out `patch` section for LDK Git dependencies
Track pending payments with their replaced/conflicting transaction IDs in a separate store. Pending payments are created here on WalletEvent::TxUnconfirmed, then removed once they reach ANTI_REORG_DELAY confirmations. This avoids scanning the entire payment store and enables efficient cleanup.
…sactions Replace the full transaction list scan in `update_payment_store` with handling of BDK's `WalletEvent` stream during sync. This leverages the new events in BDK 2.2, reduces redundant work, and prepares the foundation for reliable RBF/CPFP tracking via `WalletEvent::TxReplaced`
…events-sync Use BDK events in `update_payment_store`
Document the project architecture including core components, module organization, key design patterns, lifecycle management, and important type aliases. This provides context for AI assistants working with the codebase. Generated with Claude Code (AI-assisted) Co-Authored-By: HAL 9000 Signed-off-by: Elias Rohrer <dev@tnull.de>
Add basic `CLAUDE.md` file
Previously, the timeouts applied to chain syncing weren't configurable to users. While historically there were good reasons for this (mostly to avoid leaving the Node in a blocked state for extended periods during chain syncing), by now we should be good to let the users configure timeouts ~freely, if they deem it fit. Here we allow for exactly that. Signed-off-by: Elias Rohrer <dev@tnull.de>
It seems that users often hit these timeouts when running in production, especially when run in sub-optimal network condidtions. Here we considerably bump the timeouts, in the hopes that users now normally shouldn't hit them ever. Signed-off-by: Elias Rohrer <dev@tnull.de>
We bump our LDK dependency to commit 7fe3268475551b0664d315bfbc860416ca8fc774. Signed-off-by: Elias Rohrer <dev@tnull.de>
…pendency Bump LDK to latest `main`
…imeouts-configurable Make sync timeouts configurable
enigbe
force-pushed
the
2025-10-tiered-data-storage
branch
3 times, most recently
from
d1e0cda to
d3884bc
Compare
Introduces TierStore, a KVStore implementation that manages data across three storage layers: - Primary: Main/remote data store - Ephemeral: Secondary store for non-critical, easily-rebuildable data (e.g., network graph) with fast local access - Backup: Tertiary store for disaster recovery with async/lazy operations to avoid blocking primary store Adds four configuration methods to NodeBuilder: - set_tier_store_backup: Configure backup data store - set_tier_store_ephemeral: Configure ephemeral data store - set_tier_store_retry_config: Configure retry parameters with exponential backoff - build_with_tier_store: Build node with primary data store These methods are exposed to the foreign interface via additions in ffi/types.rs: - ForeignDynStoreTrait: An FFI-safe version of DynStoreTrait - FfiDynStore: A concrete wrapper over foreign language stores that implement ForeignDynStoreTrait.
This commit adds unit, integration, and FFI tests for the TierStore implementation: - Unit tests for TierStore core functionality - Integration tests for nodes built with tiered storage - Python FFI tests for foreign key-value store
remove features relating to retry configuration and reading/listing from backup store as a fallback strategy
remove the execution of locked writes from tier store & remove inappropriate kvstoresync usage
Additionally: - Replace TestStore with FilesystemStore for ephemeral store (TestStore's async writes block until manually completed, causing test hangs) - Use build_with_store_internal to share Node's runtime with TierStore instead of creating separate runtimes
enigbe
force-pushed
the
2025-10-tiered-data-storage
branch
from
3de73e6 to
72d38ea
Compare
delegate to kvstoresync for sync calls
enigbe
force-pushed
the
2025-10-tiered-data-storage
branch
from
b5e980f to
67d47c2
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What this PR does:
We introduce
TierStore, aKVStoreimplementation that manages data acrossthree distinct storage layers.
The layers are:
(e.g., network graph). This tier aims to improve latency by leveraging a
local
KVStoredesigned for fast/local access.asynchronously/lazily to avoid blocking primary store operations.
We also permit the configuration of
Nodewith these stores allowingcallers to set exponential back-off parameters, as well as backup and ephemeral
stores, and to build the
Nodewith TierStore's primary store. These configurationoptions also extend to our foreign interface, allowing bindings target to build the
Node with their own
ffi::KVStoreimplementations.A sample Python implementation is added and tested.
Additionally, we add comprehensive testing for
TierStoreby introducingTierStorecore functionality.Nodebuilt with tiered storage.ffi::KVStoreimplementations.Concerns
It is worth considering the way retry logic is handled, especially because of nested
retries.
TierStorecomes with a basic one by default but there areKVStoreimplementationsthat come with them baked-in (e.g.
VssStore), and thus would have no need forthe wrapper-store's own logic.