feat(dispatch): job lifecycle enqueue/claim/renew/reclaim with Lua (slice 2/6)

[as535364]

Slice 2/6 of the pull-based job dispatch redo (closes Normal-OJ/Normal-OJ#62). Dark PR: no production caller yet.

What

• dispatch/scripts.py — three Lua scripts, one per state transition (spec §8):
  • claim_pending: RPOP loop; destroys stale (non-current) jobs on the way (INV4), sets lease + attempts+=1 on the claimed job.
  • renew_lease: owner-only deadline extension; never creates or resurrects keys.
  • reclaim_expired: eligibility decided only by lease_deadline < now (INV2); at MAX_ATTEMPTS returns -1 and leaves the job untouched (JE landing is slice 3).
• dispatch/job.py — enqueue_job / claim_next_job / renew_lease, plus the 15s time-gated orphan scan (SET NX EX on dispatch:last_recovery). Time always comes from Python and enters Lua via ARGV (deterministic, fakeredis-friendly).
• tests/test_dispatch_job.py — 15 unit tests (fakeredis + lupa, no docker): claim happy path, INV4 currency destroy, INV2 expiry-only reclaim, double-reclaim race, INV5 attempts boundary, time-gate behavior.
• lupa added as a dev dependency (spec §16 requires fakeredis + lupa for Lua unit tests).

Deliberately deferred per spec §15.2: abort_requeue and JE landing to Mongo (slice 3), HTTP endpoints and presigned code_url (slice 4).

Notes for reviewers

• The claim script must build job:<jb> / submission:<sid>:current_job inside Lua from an id it discovers via RPOP, so those two key templates are spelled in Lua mirroring redis_keys.py (documented in the module docstring).
• The orphan scan hands at most one reclaimed job per 15s window to the calling runner (spec §7.3 shape). If one runner crash orphans several jobs at once, the rest are picked up in later windows — worth revisiting when slice 3 adds the full-walk JE sweep on the same loop.
• Verified: 15/15 new tests, full suite 556 passed / 9 skipped / 1 xfailed, yapf clean.

[Copilot]

Pull request overview

Implements slice 2/6 of the pull-based dispatch redesign by introducing Redis Lua scripts and a Python wrapper to support job enqueue/claim/renew/reclaim behavior, plus unit tests that exercise these transitions under fakeredis + Lua.

Changes:

• Added Redis Lua scripts for claim_pending, renew_lease, and reclaim_expired, registered via dispatch/scripts.py.
• Added Python API in dispatch/job.py for enqueue/claim/renew plus a time-gated orphan recovery scan.
• Added unit tests for lifecycle transitions and added lupa as a dev dependency to enable Lua execution in tests.

Reviewed changes

Copilot reviewed 4 out of 5 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
dispatch/scripts.py	Adds Lua scripts implementing atomic job state transitions used by dispatch.
dispatch/job.py	Adds Python wrapper functions for job enqueue/claim/renew and time-gated orphan recovery.
tests/test_dispatch_job.py	Adds unit tests covering the new lifecycle behavior using fakeredis + Lua.
pyproject.toml	Adds lupa dev dependency to support Lua script execution in tests.
poetry.lock	Locks the new lupa dependency and its artifacts.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[as535364]

codex:

我看完這個 slice，Lua state transition 本身沒有找到明確的 correctness 問題。有一個測試缺口建議 merge 前補上：

tests/test_dispatch_job.py 的 test_double_reclaim_first_wins_second_gets_zero 現在是先呼叫 rn_x，再呼叫 rn_y。第二次一定會看到第一次寫入的新 deadline，所以這沒有真的測到 issue #62 / spec §16 要求的「雙 runner race」。建議用兩個 caller 搭配 barrier 同步起跑，確認結果恰好是一個 1、一個 0，而且 attempts 只增加一次。

Non-blocking：job.py 目前用 primitives 接收 job data，claim_next_job 回傳 Optional[Dict]。typed coercion 已排在 slice 4，所以我不會因為這點擋這個 PR；如果後續 caller 在 slice 4 前就要接上，至少可以先補 TypedDict，把 boundary 說清楚。

本地驗證：556 passed, 9 skipped, 1 xfailed；yapf 和 poetry check --lock 都通過。