/track return 500 error

[luminous8]

After a Redis restart (e.g. container recreation during an update), the API starts returning 500 errors on every /track request. No events are tracked — all incoming analytics data is silently lost. The only recovery is to manually restart the API container. In my case this went unnoticed for ~5 days, resulting in a complete gap in analytics data.

Environment

Component	Version
OpenPanel API	lindesvard/openpanel-api:2.2.0
OpenPanel Dashboard	lindesvard/openpanel-dashboard:2.2.0
OpenPanel Worker	lindesvard/openpanel-worker:2.2.0
Git commit	0d1773eb (self-hosting tag)
Redis	7.2.5-alpine
ClickHouse	25.10.2.65
PostgreSQL	14-alpine
Docker	28.5.2
Docker Compose	v2.40.3
OS	Ubuntu 24.04.3 LTS
Kernel	6.8.0-101-generic
Deployment	Self-hosted via docker-compose

Steps to Reproduce

1. Self-host OpenPanel with docker-compose
2. Send tracking events — confirm they work
3. Restart only the Redis container: docker compose restart op-kv
4. Send tracking events — all return 500
5. Check API logs — every request shows: NOSCRIPT No matching script. Please use EVAL.
6. Only fix: manually restart the API container

Root Cause

In packages/db/src/buffers/event-buffer.ts, the EventBuffer class loads Lua scripts into Redis at startup via SCRIPT LOAD and caches their SHA hashes in this.scriptShas. All subsequent calls use EVALSHA with these cached SHAs (line 303).

When Redis restarts, all loaded scripts are evicted. However, the cached SHAs in the API process remain stale. Every EVALSHA call returns a NOSCRIPT error. The error is caught by the generic try-catch at line 283-284 and only logged — the event is silently lost.

The existing EVAL fallback (lines 304-310) never triggers because it only runs when sha is falsy. After a successful loadScripts() at startup, sha is always truthy — even after Redis loses the script:

// packages/db/src/buffers/event-buffer.ts, lines 299-311
const sha = this.scriptShas[scriptName];
if (sha) {
  // Always takes this branch after startup — even if Redis lost the script
  multi.evalsha(sha, numKeys, ...args);
} else {
  // This fallback never executes once scripts have been loaded
  multi.eval(scriptContent, numKeys, ...args);
  this.logger.warn(`Script ${scriptName} not loaded, using EVAL fallback`);
  this.loadScripts();
}

Impact

• All tracking events are silently dropped after any Redis restart
• No automatic recovery — requires manual API restart
• The error is logged as Failed to add event to Redis buffer but easily missed
• In my case, this resulted in ~5 days of lost analytics data before noticing

Note

This was flagged by the automated code reviewer on PR #231 ("Feature/performance2") but was merged with the concern unresolved.

Suggested Fix

After multi.exec(), check results for NOSCRIPT errors, clear stale SHAs, and retry:

const results = await multi.exec();
if (results?.some(([err]) => err?.message?.includes('NOSCRIPT'))) {
  this.scriptShas = {} as any;
  await this.loadScripts();
  // Retry with EVAL fallback
}

Or use ioredis's built-in defineCommand() which handles NOSCRIPT fallback automatically.