You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Add RedisBloom-compatible CF.DEL key item support for Cuckoo Filter.
CF.DEL deletes one matching fingerprint occurrence and returns 1 when a slot is cleared, or 0 when the key/item
is not found. Duplicate inserts require duplicate deletes, matching RedisBloom behavior.
Design
The command layer adds cf.del as a write command and delegates deletion to CuckooChain::Delete.
Deletion loads the Cuckoo Filter metadata, hashes the item, generates the fingerprint, then searches sub-filters from
newest to oldest. The first matching slot in either candidate bucket is cleared, and only one occurrence is removed.
After a successful delete, metadata size is decremented and num_deleted_items is incremented. When the chain has
more than one sub-filter and accumulated deletes exceed 10% of the remaining item count, an internal compact pass is
triggered. Compact tries to move fingerprints from newer sub-filters into older ones, removes fully compacted latest
sub-filters, and deletes their persisted page keys to avoid stale data if the chain expands again later.
Page key construction is shared through small Cuckoo page helpers so compact cleanup uses the same encoding as normal
page access.
This pr is This PR was written using codex and GPT-5.5
Hi @jihuayu, while implementing CF.DEL, I found that the automatic compaction path may introduce serious worker-blocking and OOM risks. Currently, it scans all logical buckets of a sub‑filter and loads every page into CuckooPageCache, including zero‑filled pages for missing keys. Thus, a large sparse sub‑filter can make a single CF.DEL perform work and allocate memory proportional to its logical capacity.
We seek feedback on whether compaction should be part of this PR. Two options:
Land CF.DEL without automatic compaction for now.
Only remove fingerprint and update metadata. Semantics are preserved, but deleted slots and extra sub‑filters remain, causing space and lookup amplification.
Move compaction to bounded, incremental background maintenance.
Use Kvrocks’ TaskRunner (used by AsyncCompactDB, AsyncScanDBSize, etc.) with cursor‑based scanning. Process a bounded number of actually stored pages per slice, commit a bounded write batch, release cache and locks, and requeue if more work remains. Requires actual‑page iteration, bounded cache/batch, deduplication, synchronization, and crash‑safe progress tracking.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
part of: #3552
Summary
Add RedisBloom-compatible
CF.DEL key itemsupport for Cuckoo Filter.CF.DELdeletes one matching fingerprint occurrence and returns1when a slot is cleared, or0when the key/itemis not found. Duplicate inserts require duplicate deletes, matching RedisBloom behavior.
Design
The command layer adds
cf.delas a write command and delegates deletion toCuckooChain::Delete.Deletion loads the Cuckoo Filter metadata, hashes the item, generates the fingerprint, then searches sub-filters from
newest to oldest. The first matching slot in either candidate bucket is cleared, and only one occurrence is removed.
After a successful delete, metadata size is decremented and
num_deleted_itemsis incremented. When the chain hasmore than one sub-filter and accumulated deletes exceed 10% of the remaining item count, an internal compact pass is
triggered. Compact tries to move fingerprints from newer sub-filters into older ones, removes fully compacted latest
sub-filters, and deletes their persisted page keys to avoid stale data if the chain expands again later.
Page key construction is shared through small Cuckoo page helpers so compact cleanup uses the same encoding as normal
page access.
This pr is This PR was written using codex and GPT-5.5