Skip to content

fix: release per-key coalescing lock when RTCache writer is cancelled - #948

Open
stareezy-1 wants to merge 2 commits into
cloudflare:mainfrom
stareezy-1:fix/931-rtcache-lock-leak
Open

fix: release per-key coalescing lock when RTCache writer is cancelled#948
stareezy-1 wants to merge 2 commits into
cloudflare:mainfrom
stareezy-1:fix/931-rtcache-lock-leak

Conversation

@stareezy-1

Copy link
Copy Markdown

Summary

Fixes #931

Problem

RTCache::get() inserts a zero-permit CacheLock into lockers on a cache miss, then awaits the user-provided Lookup::lookup() callback. The lock was only released and removed after the lookup completed:

let value = CB::lookup(key, extra).await;   // cancellation point
// ... cleanup below never runs if this future is dropped
my_write.lock.add_permits(10);
lockers.remove(&hashed_key);

If the get() future is cancelled while the callback is pending (timeout, losing select! branch, task abort, runtime shutdown), the lock stays installed with zero permits and no live owner. Later same-key requests then:

  • block forever with lock_age = None / lock_timeout = None (per-key liveness failure), or
  • repeatedly bypass normal coalescing / cache insertion with lock_timeout / lock_age set.

Fix

Introduce a WriterLockGuard that owns the inserted lock entry:

  • On drop — including cancellation of the writer future — it wakes waiters (add_permits) and removes the lock from lockers.
  • Removal is guarded by an Arc identity check so a cancelled old writer never deletes a newer writer's lock.
  • The normal completion path now relies on the same guard's drop, so there is exactly one cleanup path.

lockers becomes Arc<RwLock<...>> so the guard can reach it without borrowing the cache.

Tests

test_cancelled_lookup_releases_coalescing_lock (inverted from the issue's whitebox repro):

Step Verifies
Writer enters lookup() then is aborted cancellation window reachable
lockers no longer contains the key cancelled writer removes its lock — fails without the fix
Next same-key get() completes with a fresh lookup bounded progress; second call becomes the new writer

Full crate suite passes (cargo test -p pingora-memory-cache), cargo fmt --check and clippy clean.

A cache-miss writer inserts a zero-permit CacheLock before awaiting the
user Lookup callback, and only removed it after the lookup completed. If
the get() future was cancelled while the callback was pending, the lock
remained installed with zero permits and no live owner, blocking all
later same-key requests indefinitely (or bypassing normal coalescing with
lock_timeout/lock_age).

Introduce a WriterLockGuard that owns the inserted lock entry: on drop,
including cancellation of the writer future, it wakes waiters and removes
the lock from the lockers map, guarded by an Arc identity check so a
cancelled old writer never deletes a newer writer's lock.
Refactoring the writer cleanup into the guard's Drop left a bare
'let ret = ...; ret' tail, which clippy::let_and_return rejects under
-D warnings on Rust 1.91.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

RTCache lookup cancellation can leave same-key requests blocked on a stale coalescing lock

1 participant