Worker: silent shard-stuck when `worker.run()` fails silently during startup

[ayushjhanwar-png]

Summary

apps/worker/src/boot-workers.ts calls worker.run() for each GroupMQ event shard without await or .catch(). If the underlying redis.duplicate() blocking-client setup silently fails for a specific shard (e.g., a transient Redis blip during a rolling-deploy connection burst), the promise rejects with no handler and the worker enters GroupMQ's exponential backoff sleep forever. The pod appears healthy, "Started worker for events_N" is logged, but that shard never consumes.

Impact (real production incident, 2026-07-09)

• 3 of 24 event shards silently stopped consuming after a rolling deploy
• 570K jobs accumulated in Redis over ~2 hours before it was noticed
• Redis memory 2% → 22% (~9 GB)
• Cascaded to API pod OOM from increased Redis load
• Zero error logs anywhere in the worker code path
• Manual rolling-restart of the StatefulSet resolved it (fresh redis.duplicate() succeeded on the second try)

Root cause

apps/worker/src/boot-workers.ts around line 170:

worker.on('completed', markEventsActivity);
worker.on('drained', markEventsActivity);

worker.run();                                                  // ← unawaited, no .catch()
workers.push(worker);
logger.info({ concurrency }, `Started worker for ${queueName}`);

Right below this, the Kafka consumer startup is handled correctly:

startKafkaEventsConsumer()
  .then((h) => { handle = h; })
  .catch((err) => { ... });

So the GroupMQ path is inconsistent with how Kafka is handled.

Introduced in da59622d (2025-11-15, "fix: overall perf improvements"). Has been latent for ~8 months.

Suggested fix

.catch() on worker.run() — crash the pod on startup failure so k8s restarts with fresh Redis connections. Also add a worker.on('error', ...) listener so runtime errors on the blocking client are visible.

Happy to open a PR — will follow up shortly.