Skip to content

feat: extend the request lock from extendTimeout - #4041

Open
atirna wants to merge 1 commit into
apify:masterfrom
atirna:fix/extend-request-lock
Open

feat: extend the request lock from extendTimeout#4041
atirna wants to merge 1 commit into
apify:masterfrom
atirna:fix/extend-request-lock

Conversation

@atirna

@atirna atirna commented Aug 15, 2026

Copy link
Copy Markdown

Description

I added an optional per-request lock prolongation to the storage contract, prolongRequestLock(requestId, secs), and wired it to context.extendTimeout, so a request that asks for more time does not have its lock expire on a locking storage backend while it is still being processed.

Previously extendTimeout pushed the handler window, the navigation window and the internal timeout, but not the storage lock. Raising setExpectedRequestProcessingTimeSecs from extendTimeout was tried in #3860 and reverted (the hint is process-wide, raise-only, and not retroactive), so this goes per-request instead, as proposed in #3961:

  • RequestQueueBackend.prolongRequestLock?(requestId, secs) — optional, backends without locking leave it unimplemented; resolves false when the lock could not be prolonged (information, not an error)
  • RequestQueue, RequestManagerTandem and ThrottlingRequestManager forward it (the throttling one routes to the manager holding the request without consuming the in-flight marker used by markRequestAsHandled/reclaimRequest)
  • context.extendTimeout fires it best-effort: a rejection is logged at debug level, it never fails the request that asked for more time

Fixes #3961

Why

On a locking storage backend (e.g. the Apify request queue) the lock is sized once from the expected processing time. A request that extends its timeout past that hint can outlive its lock, and another consumer sharing the queue may get the same request handed out while the first one is still processing it.

Verification

  • new tests: frontend forwarding (true/false cases), tandem forwarding, throttling routing (marker preserved, request then marked handled in the right manager), and the crawler wiring (context.extendTimeout prolongs the lock of the very request the handler runs for)
  • sabotage check: removing the wiring makes the crawler test fail (prolonged stays empty), restoring it passes again
  • pnpm vitest run on the four touched test files: 224 passed
  • pnpm tsc-check-tests, pnpm lint (0 errors), pnpm format:check, pnpm api:check all clean; docs/public-api reports regenerated

@janbuchar
janbuchar self-requested a review August 17, 2026 14:03
@B4nan
B4nan changed the base branch from v4 to master August 18, 2026 16:33
Adds an optional per-request lock prolongation to the request queue backend
contract and wires it to `context.extendTimeout`, so a request that asks for
more time does not have its lock expire on a locking storage backend while it
is still being processed.

Fixes apify#3961
@B4nan
B4nan force-pushed the fix/extend-request-lock branch from e39cd6e to de06345 Compare August 18, 2026 18:25
@B4nan

B4nan commented Aug 18, 2026

Copy link
Copy Markdown
Member

heads up: the v4 branch was merged into master (v3 development moved to the 3.x branch), so I rebased this branch onto master and force-pushed it to your fork. Your commits are unchanged, just replayed on the new base. Before adding more changes, reset your local branch to the remote one (e.g. git fetch && git reset --hard @{u}), otherwise you'd reintroduce the old history.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

extendTimeout should extend the request lock on locking storage backends

3 participants