[scheduler/cuebot] Replace Redis-backed accounting with in-memory store + PG LISTEN/NOTIFY

[DiegoTavares]

The accounting subsystem coordinated Cuebot and the Rust scheduler through Redis to enable horizontal scaling across N scheduler instances. The scheduler is and will remain single-instance (N=1), so Redis's only unique benefit is unreachable while it manufactured an entire class of accounting-drift bugs (limit-seeding fail-closed, mass dispatch rejection, double-booking, CAS starvation). A single in-process counter is the source of truth that makes that bug class structurally impossible.

All shows were already drained to Cuebot-managed, so the cutover is clean (deploy into an idle role, then flip shows back one at a time via b_scheduler_managed).

Scheduler — Redis → in-memory Store:

• accounting/store.rs: one Mutex, atomic check-and-increment across the three enforced vertices (subscription burst, folder/job max cores+gpus). Layer/point were incremented but never read, so they're dropped.
• Live updates via PG LISTEN/NOTIFY (accounting/listener.rs): acct_release and acct_limit_change.
• Recompute from SUM(proc) is the backstop (absolute overwrite, no CAS), carrying in-flight bookings forward via an epoch double-buffer so it can never erase a not-yet-snapshot-visible booking → never over-books a hard cap.
• Blocking seeds gate dispatch: bootstrap and managed-flip both seed caps and booked counters before enforcing.
• Deleted redis_client.rs, lua.rs, acct:seq/CAS, the redis dependency.

Cuebot — LettuceAccountingRedisPublisher → AccountingNotifier:

• Transactional pg_notify on proc release (same txn as DELETE proc → delivered iff it commits, a stronger model than the old afterCommit publish) and on the five enforced admin cap changes.
• accounting.redis.* and the Lettuce dep removed; replaced by a safe accounting.notify.enabled kill-switch (off → scheduler degrades to recompute-only, which under-books, never over-books). The old over-booking startup guardrail is gone.

Docs — redis-accounting.md → scheduler-accounting.md (full rewrite) plus scheduler.md, deploying-scheduler.md, stress-testing, and properties.

• Caps are hard (license/OOM); every failure mode is safe-direction: a dropped NOTIFY leaves a counter reading high → under-book → healed by the next recompute.

• N=1 is now an assumption of the in-memory design; multi-scheduler would need a shared store again (revisit trigger documented).

• Rust: 177 lib tests pass (incl. straddle, managed-flip, dropped-NOTIFY invariants); clippy clean; stress suite compiles; bin builds.

• Cuebot: compileJava/compileTestJava/spotlessJavaCheck pass (JDK 11). rollback).

[coderabbitai]

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 89aff33b-f80a-4851-8acc-beb193255f39

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}