Skip to content

fix(repo_store): use SHA-256 for stable advisory-lock key (#210) - #215

Open
Gravirei wants to merge 4 commits into
Gitlawb:mainfrom
Gravirei:fix/issue-210-stable-advisory-lock-key
Open

fix(repo_store): use SHA-256 for stable advisory-lock key (#210)#215
Gravirei wants to merge 4 commits into
Gitlawb:mainfrom
Gravirei:fix/issue-210-stable-advisory-lock-key

Conversation

@Gravirei

@Gravirei Gravirei commented Jul 17, 2026

Copy link
Copy Markdown
Contributor

Summary

Replace DefaultHasher with SHA-256 for the Postgres advisory-lock key so the same (owner_slug, repo_name) produces the same i64 across all Rust toolchain versions, OSes, and machines — fixing silent cross-machine write exclusion breakage.

Motivation & context

Closes #210

Kind of change

  • Bug fix

What changed

  • gitlawb-node: advisory_lock_key() in repo_store.rs now uses sha2::Sha256 instead of std::collections::hash_map::DefaultHasher (not guaranteed stable across Rust releases).
  • gitlawb-node: acquire_write doc comment scoped to clarify cross-machine guarantee holds only on shared Postgres.
  • gitlawb-node: Golden-value unit test pins the key for a known input so CI catches accidental algorithm changes.

How a reviewer can verify

cargo test -p gitlawb-node -- git::repo_store::tests::advisory_lock_key_is_stable

Before you request review

  • Scope is one logical change; no unrelated churn
  • cargo test --workspace passes locally (relevant tests pass)
  • New behavior is covered by tests
  • cargo fmt --all and cargo clippy --workspace --all-targets -- -D warnings are clean
  • Commit titles use Conventional Commits (fix(...))

Protocol & signing impact

N/A — internal implementation detail, no protocol change.

Notes for reviewers

Changing the algorithm re-keys every node. During a mixed-version upgrade window, an in-flight lock held under the old key will not exclude a new-key caller, so this needs a staged rollout (or an accepted brief window), not a hot swap.

Summary by CodeRabbit

  • Bug Fixes
    • Improved repository write locking to retain the same PostgreSQL connection for advisory-lock acquisition and release.
    • Ensured advisory locks are explicitly unlocked when write acquisition fails during repository preparation/fallback paths.
    • Made advisory-lock keys deterministic (SHA-256 of owner + repo), and ensured locks are released when the guard is dropped.
  • Documentation
    • Updated the changelog with “Unreleased” known caveats and rolling-upgrade guidance for the lock-key change.

Copilot AI review requested due to automatic review settings July 17, 2026 16:04

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@coderabbitai

coderabbitai Bot commented Jul 17, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

Warning

Review limit reached

@Gravirei, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 34 minutes

Your organization has reached its usage spending cap. Adjust your spending cap in the billing tab.

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 804df2cf-10b5-4e53-987d-90df478c56e5

📥 Commits

Reviewing files that changed from the base of the PR and between 26c4dfe and 30e0dec.

📒 Files selected for processing (5)
  • crates/gitlawb-node/src/auth/mod.rs
  • crates/gitlawb-node/src/git/repo_store.rs
  • crates/gitlawb-node/src/main.rs
  • crates/gitlawb-node/src/test_support.rs
  • docs/RUN-A-NODE.md
📝 Walkthrough

Walkthrough

Repository advisory locks now retain their acquiring PostgreSQL connection, explicitly unlock on a fallback error path, and derive keys deterministically with SHA-256. Tests cover stable and input-sensitive keys, while the changelog documents mixed-version rollout caveats.

Changes

Repository advisory-lock handling

Layer / File(s) Summary
Lock connection lifecycle
crates/gitlawb-node/src/git/repo_store.rs
acquire_write() retains the connection that acquires the advisory lock, stores it in RepoWriteGuard, and explicitly unlocks it on the Tigris-download fallback error path.
Guard release and deterministic key contract
crates/gitlawb-node/src/git/repo_store.rs, CHANGELOG.md
RepoWriteGuard releases and drop-closes the retained connection; advisory-lock keys use SHA-256 over repository identity, with stability and input-variation tests and documented rolling-upgrade caveats.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Sequence Diagram(s)

sequenceDiagram
  participant RepoStore
  participant Postgres
  participant RepoWriteGuard
  RepoStore->>Postgres: Acquire dedicated connection
  RepoStore->>Postgres: Try advisory lock
  Postgres-->>RepoStore: Return lock result
  RepoStore->>RepoWriteGuard: Store connection and lock key
  RepoWriteGuard->>Postgres: Unlock through retained connection
Loading

Possibly related PRs

  • Gitlawb/node#54: Overlaps the repository lock acquisition, fallback, and release paths.
  • Gitlawb/node#196: Its purge workflow uses the per-repository advisory locks updated here.
🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Title check ✅ Passed The title concisely matches the main change: a stable SHA-256 advisory-lock key for repo_store.
Description check ✅ Passed The description covers the required sections, verification command, checklist, and rollout notes.
Linked Issues check ✅ Passed The changes satisfy #210 with SHA-256 keying, a golden test, scoped docs, and rollout caveats.
Out of Scope Changes check ✅ Passed No clearly unrelated changes stand out; the lock-lifecycle edits remain tied to advisory-lock correctness.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@beardthelion beardthelion added crate:node gitlawb-node — the serving node and REST API kind:bug Defect fix — wrong or unsafe behavior labels Jul 17, 2026

@beardthelion beardthelion left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The fix is correct and the golden test is load-bearing. I verified the pinned value is the real SHA-256 (recomputed SHA-256("did_key_...:hello")[..8] as LE i64 == -6680856138670956537, and big-endian would differ, so byte order is pinned), then ran a mutation matrix at the head: reverting to DefaultHasher, flipping the separator, flipping to from_be_bytes, and dropping owner_slug from the hash each turn the golden test RED, and it's GREEN restored. The separator is injective because owner_slug is colon-sanitized upstream (owner_did.replace([':','/'], "_")), so there's no wrong-repo aliasing, and the byte-slice path can't panic. This closes #210 the way it asked.

One thing to resolve before it lands, plus a small test note.

Findings

  • [P2] Persist the re-keying rollout caveat where operators will see it after merge
    crates/gitlawb-node/src/git/repo_store.rs:407
    Switching the algorithm re-keys every node. On a shared-Postgres, multi-node deployment doing a rolling upgrade, an old-binary node (DefaultHasher key) and a new-binary node (SHA-256 key) compute different keys for the same repo and stop mutually excluding for the deploy window, a transient loss of the cross-machine write-exclusion this lock exists to provide (concurrent same-repo pushes could race). That topology is real (the shared-DB compose target, plus purge-spam as a separate process on the same lock namespace). Right now the caveat lives only in the PR body, which disappears on merge. I filed #210 with "a staged rollout OR an accepted brief window" as acceptable, so I'm not asking you to change the swap itself, only to make the guidance durable: a short note on acquire_write (drain writes or cut over single-node during the upgrade) plus a CHANGELOG line. If you'd rather remove the window entirely, the mechanism is a transition release where acquire_write takes both the legacy and the new key, dropping the legacy one a release later, but that's optional given the accepted-window path.

  • [P3] Strengthen or drop advisory_lock_key_differs_for_different_inputs
    crates/gitlawb-node/src/git/repo_store.rs:594
    It varies both inputs at once (owner_a/repo_a vs owner_b/repo_b), so it can't isolate a regression that drops one parameter, and I confirmed that by mutation: hashing only repo_name leaves this test green while the golden test goes red. So the golden test already backstops that case and this one adds little. If you want it to pull weight, assert a same-owner/different-repo pair and a different-owner/same-repo pair; otherwise it's fine to drop it and lean on the golden test.

Not a change request: the Cargo.lock version bumps (0.5.0 -> 0.5.1 across five crates, plus a sha2 line under icaptcha-client) aren't from this PR. The manifests are already 0.5.1 on main and gitlawb-node already declares sha2; the lockfile was just stale (that's issue #185), and regenerating it here corrected the drift. Leave them as they are; a one-line mention in the PR description is enough.

Solid, well-scoped fix. Once the rollout note is durable I'm happy to approve.

@jatmn jatmn left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I found issues that need to be addressed before this is ready.

Findings

  • [P1] Retain the lock-owning database session for the write guard
    crates/gitlawb-node/src/git/repo_store.rs:164
    This change makes the key deterministic, but pg_try_advisory_lock is still executed through &PgPool and the guard later unlocks through that pool again. Those are session-scoped PostgreSQL locks: SQLx returns the acquiring connection after fetch_one, so release() can be assigned another session and silently fail to unlock. Worse, a later writer can be given the idle lock-owning session and reentrantly acquire the same lock. The advertised shared-Postgres exclusion therefore remains unreliable even with the SHA-256 key. Hold a PoolConnection<Postgres> in RepoWriteGuard from successful acquisition through unlock (including error cleanup), and add a competing-session test. PR #196 already has this exact dedicated-connection repair, so either make this PR depend on/rebase onto it or bring that repair here before merging.

  • [P2] Preserve exclusion while old and new lock keys coexist
    crates/gitlawb-node/src/git/repo_store.rs:407
    Replacing DefaultHasher re-keys every repository. In a rolling shared-Postgres upgrade, an old node can hold the legacy key while a new node acquires the SHA-256 key for the same repository; PostgreSQL treats them as independent locks, allowing concurrent receive-pack/issue/pull writes and competing archive uploads. The PR body acknowledges this window, but it disappears on merge and the checked-in documentation only describes the topology. Acquire both keys for a transition release, or add durable deployment guidance requiring writes to be drained or cut over through a single node before versions are mixed.

Address review findings on Gitlawb#210:
- RepoWriteGuard now stores PoolConnection<sqlx::Postgres> instead of
  PgPool, so the pg_advisory_unlock on release() runs on the same
  session that acquired the lock (no silent no-op unlock, no
  reentrant acquisition by a later writer handed the idle
  lock-owning connection). Error path also releases the lock on the
  same connection before returning.
- Document the rolling-upgrade re-keying caveat durably on
  acquire_write (drain writes or cut over single-node during upgrade)
  and in CHANGELOG's Unreleased "Known caveats" section, since the PR
  body disappears on merge.
- Strengthen advisory_lock_key_differs_for_different_inputs to vary
  one axis at a time (same-owner/diff-repo, diff-owner/same-repo) so a
  regression that drops one parameter from the hash is caught, not
  just one that drops both.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)
crates/gitlawb-node/src/git/repo_store.rs (1)

410-457: 🩺 Stability & Availability | 🔴 Critical | ⚡ Quick win

Task cancellation leaks session advisory locks to the connection pool.

If an async task is cancelled (e.g., due to a client timeout or disconnect) while holding the lock, the sqlx::pool::PoolConnection is implicitly dropped and returned to the pool. Because Postgres session-level advisory locks are not cleared automatically when a connection is returned to the pool, any subsequent request drawing this connection will inherit the lock. This can silently lock out legitimate writers or grant unintended access. This hazard applies both to downstream callers holding the guard (e.g., while awaiting receive_pack as seen in snippet 1) and within acquire_write itself during the tigris.download .await.

  • crates/gitlawb-node/src/git/repo_store.rs#L410-L457: Wrap conn in an Option and implement Drop for RepoWriteGuard. If the guard is dropped before release() is explicitly called, invoke conn.detach() to discard the connection entirely, which forces the Postgres session to close and safely clears the lock.
  • crates/gitlawb-node/src/git/repo_store.rs#L231-L266: Construct RepoWriteGuard immediately after lock acquisition so its new Drop implementation protects the tigris.download(...).await point against task cancellations.
🔒️ Proposed fix to implement cancellation safety

For crates/gitlawb-node/src/git/repo_store.rs#L410-L457:

 pub struct RepoWriteGuard {
     owner_slug: String,
     repo_name: String,
     pub local_path: PathBuf,
     lock_key: i64,
-    conn: PoolConnection<sqlx::Postgres>,
+    conn: Option<PoolConnection<sqlx::Postgres>>,
     tigris: Option<TigrisClient>,
 }
 
+impl Drop for RepoWriteGuard {
+    fn drop(&mut self) {
+        if let Some(conn) = self.conn.take() {
+            // Task was cancelled before release() could run.
+            // Detach to discard the connection and close the Postgres session.
+            let _ = conn.detach();
+        }
+    }
+}
+
 impl RepoWriteGuard {
     pub fn path(&self) -> &Path {
         &self.local_path
     }
 
     pub async fn release(mut self, success: bool) {
         if success {
             if let Some(ref tigris) = self.tigris {
                 if let Err(e) = tigris
                     .upload(&self.owner_slug, &self.repo_name, &self.local_path)
                     .await
                 {
                     warn!(repo = %self.repo_name, err = %e, "failed to upload repo to tigris after write");
                 }
             }
         } else {
             warn!(repo = %self.repo_name, "write failed — skipping tigris upload to avoid propagating an inconsistent repo");
         }
 
-        let _ = sqlx::query("SELECT pg_advisory_unlock($1)")
-            .bind(self.lock_key)
-            .execute(&mut *self.conn)
-            .await;
+        if let Some(mut conn) = self.conn.take() {
+            let _ = sqlx::query("SELECT pg_advisory_unlock($1)")
+                .bind(self.lock_key)
+                .execute(&mut *conn)
+                .await;
+        }
     }
 }

For crates/gitlawb-node/src/git/repo_store.rs#L231-L266:

-        // Always download the latest from Tigris before writing.
-        // Local disk may be stale if another machine pushed since our last access.
-        if let Some(ref tigris) = self.tigris {
-            if tigris.exists(&owner_slug, repo_name).await.unwrap_or(false) {
-                debug!(repo = %repo_name, "write acquire: downloading latest from tigris");
-                if let Err(e) = tigris.download(&owner_slug, repo_name, &local_path).await {
-                    // Same self-healing fallback as acquire_fresh: a corrupt/unreadable
-                    // Tigris archive must not block a write when a valid local copy
-                    // exists — release(success) will re-upload a good archive.
-                    if local_path.exists() {
-                        warn!(repo = %repo_name, err = %e,
-                            "write acquire: tigris download failed — falling back to local copy");
-                    } else {
-                        // Drop the lock on the error path before returning; the
-                        // connection is released to the pool by Drop on
-                        // PoolConnection, but the lock must be released explicitly
-                        // so a subsequent writer isn't blocked until idle timeout.
-                        let _ = sqlx::query("SELECT pg_advisory_unlock($1)")
-                            .bind(lock_key)
-                            .execute(&mut *conn)
-                            .await;
-                        return Err(e).context("downloading repo from tigris for write");
-                    }
-                }
-            }
-        }
-
-        Ok(RepoWriteGuard {
+        let mut guard = RepoWriteGuard {
             owner_slug,
             repo_name: repo_name.to_string(),
             local_path,
             lock_key,
-            conn,
+            conn: Some(conn),
             tigris: self.tigris.clone(),
-        })
+        };
+
+        // Always download the latest from Tigris before writing.
+        // Local disk may be stale if another machine pushed since our last access.
+        if let Some(ref tigris) = self.tigris {
+            if tigris.exists(&guard.owner_slug, &guard.repo_name).await.unwrap_or(false) {
+                debug!(repo = %guard.repo_name, "write acquire: downloading latest from tigris");
+                if let Err(e) = tigris.download(&guard.owner_slug, &guard.repo_name, &guard.local_path).await {
+                    if guard.local_path.exists() {
+                        warn!(repo = %guard.repo_name, err = %e,
+                            "write acquire: tigris download failed — falling back to local copy");
+                    } else {
+                        // Explicitly release the guard on error so the connection unlocks and returns to the pool cleanly.
+                        guard.release(false).await;
+                        return Err(e).context("downloading repo from tigris for write");
+                    }
+                }
+            }
+        }
+
+        Ok(guard)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@crates/gitlawb-node/src/git/repo_store.rs` around lines 410 - 457, Update
crates/gitlawb-node/src/git/repo_store.rs:410-457 so RepoWriteGuard stores its
PoolConnection in an Option, takes it during successful release, and implements
Drop to call detach() when release() was not completed, preventing cancellation
from returning a lock-bearing session to the pool. In
crates/gitlawb-node/src/git/repo_store.rs:231-266, construct RepoWriteGuard
immediately after advisory-lock acquisition, before awaiting tigris.download, so
its Drop implementation protects that cancellation point; preserve explicit
release behavior and lock cleanup.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Outside diff comments:
In `@crates/gitlawb-node/src/git/repo_store.rs`:
- Around line 410-457: Update crates/gitlawb-node/src/git/repo_store.rs:410-457
so RepoWriteGuard stores its PoolConnection in an Option, takes it during
successful release, and implements Drop to call detach() when release() was not
completed, preventing cancellation from returning a lock-bearing session to the
pool. In crates/gitlawb-node/src/git/repo_store.rs:231-266, construct
RepoWriteGuard immediately after advisory-lock acquisition, before awaiting
tigris.download, so its Drop implementation protects that cancellation point;
preserve explicit release behavior and lock cleanup.

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 4b4619c0-2d72-49f8-b859-8dc4678f0ec7

📥 Commits

Reviewing files that changed from the base of the PR and between ad7c2b2 and c27b355.

⛔ Files ignored due to path filters (1)
  • Cargo.lock is excluded by !**/*.lock
📒 Files selected for processing (2)
  • CHANGELOG.md
  • crates/gitlawb-node/src/git/repo_store.rs

@jatmn jatmn left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I found issues that need to be addressed before this is ready.

Findings

  • [P1] Keep advisory-lock sessions out of the shared application pool
    crates/gitlawb-node/src/git/repo_store.rs:207
    acquire_write now checks out a connection from the same Db pool used by every handler and holds it for the full retry loop, Tigris I/O, and write operation. With the default 20-connection pool, one lock holder plus 19 requests contending for that repository consume every connection while the contenders sleep and retry for up to 60 seconds; distinct slow writes can do the same for up to the 600-second git-service timeout. Unrelated database-backed requests then fail their five-second acquire timeout. Use a separate bounded pool for advisory-lock sessions, or otherwise bound lock waiters/write guards while reserving capacity for the application pool.

  • [P1] Do not return a session to the pool until its advisory unlock is known to have completed
    crates/gitlawb-node/src/git/repo_store.rs:469
    release removes the lock-owning connection from self.conn before awaiting pg_advisory_unlock. If that future is cancelled, the guard's Drop sees None, so SQLx returns the session to the pool even though PostgreSQL may still hold its session-scoped advisory lock. A later borrower of that session can reentrantly acquire the lock while other sessions remain blocked, defeating the write exclusion. The Tigris-download error cleanup has the same unsafe outcome when its ignored unlock query fails. Keep the connection in the guard through a successful unlock, and close it rather than pool it on cancellation or unlock failure; add a regression test that aborts during unlock.

@beardthelion beardthelion left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-reviewed at 26c4dfe8. Both points from my last round landed properly. The rolling-upgrade caveat is now in the acquire_write docstring, and the differs-test rewrite pulls real weight: I ran the mutation matrix and dropping either parameter from the hash turns it RED (key must depend on repo_name / key must depend on owner_slug), where the old both-vary-at-once shape stayed green. The golden constant still catches an algorithm swap. The SHA-256 change itself remains correct.

Holding the lock-owning connection in the guard is the right move and the new Drop works: I confirmed that dropping a guard without calling release() frees the lock to an independent session, because close_on_drop really does end the session in sqlx 0.8.6. What is not right yet is everything that happens once the connection leaves the guard, plus a collision with #79 that will bite whichever of the two merges second.

Findings

  • [P1] Settle one lock-key derivation with #79 before either merges
    crates/gitlawb-node/src/git/repo_store.rs:496
    #79 also claims to fix #210, and it hashes format!("{owner}/{repo}") folded big-endian while this PR hashes owner + ":" + repo folded little-endian. Same repo, different key. Whichever lands second re-keys every repository a second time, breaks the other's golden constant, and makes operators pay the exclusion-gap window twice; #79's legacy fallback is main's DefaultHasher, so its dual-key transition would not cover a fleet already running this key either. My call as lead: this PR owns the lock layer, since it is the focused change that closes #210 and it got here first. I am asking #79 to drop its lock work and rebase onto this, so nothing here needs to change except knowing the decision is settled.

  • [P1] Stop pinning a pool connection for the whole 60-second acquire wait
    crates/gitlawb-node/src/git/repo_store.rs:207
    The connection is now checked out before the retry loop, so a writer that waits out a contended repo holds one of the pool's 20 connections (config.rs:192) for up to 60 seconds without doing any work. On the base commit the loop used fetch_one(&self.pool) and returned its connection between probes, so this is a regression the fix introduces rather than an existing gap. What makes it more than a tuning question is who can reach it: create_issue sits behind authorize_repo_read and nothing stronger (api/issues.rs:46) before it calls acquire_write at :66, so any authenticated identity with read access to a repo can occupy a connection for the length of a write. Only the winning attempt needs session affinity, so acquire inside the loop and drop before each sleep; I compiled that change. The other half, every in-flight write pinning a connection for its whole duration, is what jatmn's dedicated-pool ask is really about, and that one still stands.

  • [P1] Keep the connection inside the guard across the unlock await, and check the unlock result
    crates/gitlawb-node/src/git/repo_store.rs:469
    self.conn.take() moves the connection out before the await, which disarms the Drop you just added: from that instant the lock is held by a connection nothing will close. Cancellation there, or an unlock that does not actually unlock, returns that connection to the pool with the lock still held. This is worse than a one-off leak because advisory locks are counted per session. I checked it against Postgres directly: two pg_try_advisory_lock calls on one session both return true, and after a single pg_advisory_unlock the session still holds the lock. So the next writer for that repo, handed the same pooled connection, re-acquires reentrantly and its own release() only decrements the count back to one. The lock then never clears until sqlx recycles the connection, which on the 0.8.6 defaults is 10 minutes idle or 30 minutes lifetime, and every writer on every node fails the 60-second retry in the meantime. Note also that pg_advisory_unlock reports "you did not hold this" as a false return with a WARNING, not an error, so let _ = ... .execute(...) cannot see it at either this site or :267. Your own error path at :266 already uses the correct borrow shape, so the two paths just need to agree. Unlocking through if let Some(ref mut conn) = self.conn, taking the connection only once the unlock is confirmed, and otherwise warning and leaving it for Drop to close, compiles and keeps the existing 20 repo_store tests green.

  • [P2] Move the operator caveat out of CHANGELOG.md
    CHANGELOG.md:3
    My last round asked for a CHANGELOG line and that was the wrong placement, so this one is on me. That file is generated by release-please (release-please-config.json, and every commit touching it is a chore(main): release), and main is already at 0.7.0, so a hand-written ## Unreleased section will sit above the next generated release forever. Keep the docstring, which is the durable half I actually wanted, and put the operator guidance in the operational checklist in docs/RUN-A-NODE.md, next to where the signed-peer rolling-upgrade posture is already documented.

  • [P3] Drop the Cargo.lock hunk when you rebase
    Cargo.lock:3303
    It was legitimate drift correction when you wrote it, but main now carries both halves already: the crates are at 0.5.1 there and icaptcha-client already lists sha2. After a rebase this hunk is empty churn.

  • [P2] Close the same gap in the acquire loop
    crates/gitlawb-node/src/git/repo_store.rs:216
    The probe query is the one await where the connection is a bare local with no guard around it. If the future is dropped while that query is in flight, the request may still have reached Postgres and taken the lock, and the connection then returns to the pool holding it, with the same counted-lock consequence as above. If you take the acquire-inside-the-loop change this shrinks to the single winning query, and wrapping that connection so its Drop closes the session until the guard takes ownership closes it entirely.

Also still open from jatmn's last round: the two regression tests, a competing-session case proving a second Postgres session is excluded while the guard is held, and one that exercises cancellation during the unlock. The second one is worth writing RED against the current take() shape so it proves the fix rather than tracking it.

@Gravirei
Gravirei requested review from beardthelion and jatmn July 27, 2026 06:30

@jatmn jatmn left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I rechecked this at 30e0dec. The dedicated lock_pool, acquire-inside-the-retry-loop change, and keeping the lock-owning connection across the unlock .await all landed correctly since my last round. The SHA-256 key and golden test still look right. What remains is the unlock/return-to-pool contract, lock-pool sizing, and the regression tests reviewers asked for.

Findings

  • [P1] Verify pg_advisory_unlock actually released the lock before returning the session to lock_pool
    crates/gitlawb-node/src/git/repo_store.rs:506
    release() still treats a successful execute() as proof the session is clean and calls self.conn.take(), returning the connection to lock_pool. PostgreSQL reports “you did not hold this lock” as a false return value, not a query error, so execute() cannot distinguish that case. Worse, session-level advisory locks are counted: if this session already held the key (for example because a prior holder returned a lock-bearing connection to the pool and this writer re-acquired reentrantly), a single pg_advisory_unlock can return true while the session still holds the lock. The next borrower of that pooled connection inherits it, and exclusion can stay broken until sqlx recycles the session. beardthelion called this out on the prior head; it is still present on both the happy path here and the Tigris error cleanup at :288. Please unlock through query_as into (bool,), treat false like a failed unlock (leave the connection for Drop/close_on_drop), and add a regression test that aborts during unlock so this cannot regress again.
  • [P1] Hard-coding lock_pool to two connections caps the whole node at two in-flight writes
    crates/gitlawb-node/src/main.rs:283
    Each RepoWriteGuard now pins a lock_pool connection for the full write path (Tigris download, git receive-pack/merge/issue work, Tigris upload, unlock) — up to git_service_timeout_secs (600s by default). With max_connections(2), a third concurrent acquire_write() anywhere on the node blocks on lock_pool.acquire() and, after the 5s db_acquire_timeout_secs, fails with acquiring connection for advisory lock instead of entering the 60×1s advisory-lock retry loop. That is a new node-wide write denial under modest concurrency, including cross-repo traffic (repos.rs, pulls.rs, issues.rs all call acquire_write). Worse, each iteration of the contended-repo retry loop calls lock_pool.acquire().await? (:226), so a same-repo waiter that is merely waiting on an advisory lock can abort after ~5s when two unrelated writes occupy both slots — the "could not acquire advisory lock after 60s" path is unreachable under pool saturation. Please size the pool for expected concurrent writers (config/env), retry pool-acquire failures inside the lock loop, or stop holding a pooled connection for the entire git/Tigris phase once the lock is acquired.
  • [P2] The competing-session and cancellation regression tests still do not protect CI
    crates/gitlawb-node/src/git/repo_store.rs:757
    beardthelion and I both asked for load-bearing Postgres tests here. The new competing_session_is_excluded_while_guard_held and cancelled_guard_releases_lock_when_dropped cases are marked #[ignore] and make_store_for_live() wires lock_pool with connect_lazy("postgres://invalid") while the #[sqlx::test] fixture provisions pool against the real test database, so even a manual run cannot pass. The ignore reason (requires DATABASE_URL) is also misleading — CI already provisions Postgres in pr-checks.yml. Even after fixing the fixture, cancelled_guard_releases_lock_when_dropped probes lock availability immediately after handle.await while Drop only calls close_on_drop(), which spawns an async session teardown in sqlx 0.8.6, so the test can pass or fail depending on scheduler timing rather than proving the lock is gone. Please point lock_pool at the same database as the test pool, drop the #[ignore], wait for the close future (or use synchronous teardown) before probing, and keep the cases in the default cargo test -p gitlawb-node path.
  • [P2] A cancelled winning lock probe can still poison lock_pool
    crates/gitlawb-node/src/git/repo_store.rs:232
    Failed probes drop their connection without holding a lock, but the one successful pg_try_advisory_lock runs on a bare PoolConnection until RepoWriteGuard is constructed immediately afterward. If the task is cancelled during that fetch_one().await after Postgres took the lock, the connection is returned to lock_pool without close_on_drop, recreating the counted-lock/reentrancy failure mode the new Drop is meant to prevent. Please wrap the winning probe connection in the same “close on drop unless explicitly handed to the guard” RAII used for cancellation safety.
  • [P3] lock_pool startup uses a single connect() with no retry
    crates/gitlawb-node/src/main.rs:287
    The main application pool goes through connect_db_with_retry, but the advisory-lock pool is created with one connect().await? immediately afterward. A transient connection failure or brief max_connections pressure after the main pool comes up aborts node startup even though migrations already succeeded. Please reuse the same retry/backoff pattern (or share the established pool with a dedicated advisory-lock semaphore) so startup is not flakier than the rest of the DB bootstrap.
  • [P3] Operator connection sizing ignores the two extra lock_pool sessions
    crates/gitlawb-node/src/main.rs:282
    Each node now opens a second pool to the same database_url with max_connections(2) on top of GITLAWB_DB_MAX_CONNECTIONS (default 20, documented in .env.example). Fleet sizing that budgets only the main pool can hit Postgres max_connections earlier than expected and contribute to the acquire failures above. Please document the extra headroom in RUN-A-NODE.md / .env.example, or make the lock-pool size configurable alongside db_max_connections.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

crate:node gitlawb-node — the serving node and REST API kind:bug Defect fix — wrong or unsafe behavior

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Advisory-lock key uses DefaultHasher — cross-machine write exclusion can silently break across node builds

4 participants