fix(actions): do not run scheduled actions before they are due (#229) - #231
Open
nogumaruo wants to merge 1 commit into
Open
fix(actions): do not run scheduled actions before they are due (#229)#231nogumaruo wants to merge 1 commit into
nogumaruo wants to merge 1 commit into
Conversation
Cron jobs are registered in the in-memory scheduler of a single Node process, so `removeCronJob`/`scheduleCronJob` on update only take effect on the instance that handled the request. In a multi-instance deployment every other instance keeps the job it registered from the previous `executeAt`, fires at the old time and publishes the entry earlier (or later) than scheduled. Because `toggle` writes `publishedAt: record.executeAt` and the action row is deleted afterwards, the entry also goes live with a future timestamp and the intended schedule never runs at all. - re-check `currentAction.executeAt` when the job fires and skip execution when the action is not due yet (60s tolerance for clock jitter), then reschedule it with the stored value so the instance self-heals. This restores the `executeAt <= now` guarantee that the 1.x polling cron had. - only log "Removed cron job" when a job was actually removed. `strapi.cron.remove` does not throw for an unknown task name, so the previous unconditional success log hid the fact that a stale job could still live on another instance, which made this very hard to diagnose. Fixes pluginpal#229 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes #229 — a scheduled action can fire at an outdated time on instances that did not handle the reschedule request, publishing the entry before (or after) the time the editor picked.
This implements suggested fixes 1 and 2 from the issue. Suggested fix 3 (restoring a DB-polling reconciliation loop, which would also address #223) is intentionally left out of this PR so it can be discussed separately — it is a larger design change, while the two changes here are contained and make the current design safe.
The problem in one paragraph
strapi.cronis the in-memory scheduler of a single Node process.scheduleCronJobandremoveCronJobtherefore only affect the instance that happened to servePUT /publisher/actions/:id. Every other instance keeps the job it registered earlier, from the previousexecuteAt. When that stale job fires, the task re-fetches the action from the database — so it correctly sees the newexecuteAt— but then executes anyway. Two further effects make this worse than a simple mistimed publish:publicationService.togglewritespublishedAt: record.executeAt, so the entry goes live carrying a timestamp that is still in the future; and the action row is deleted afterwards, so the legitimate job on the other instance later logsAction ... no longer exists, skipping executionand the intended schedule never happens.What changes
1.
server/services/action-service.js— don't execute an action that is not due yetWhen the job fires, the freshly fetched
currentAction.executeAtis compared against the current time. If it is still more thanSTALE_JOB_TOLERANCE_MS(60s) in the future, the action is not executed; instead the stale job is dropped and rescheduled from the stored value, so the instance repairs its own state.The 60s tolerance is there so that ordinary scheduling jitter or small clock differences never cause a legitimate execution to be skipped, while the failure this addresses (minutes to days off) is always caught.
Rescheduling is deliberately done in
setImmediateinsidefinally, afterstrapi.cron.remove(taskName). Registering the new job before the removal would make the removal delete the job that was just added, since both use the same task name.This restores the guarantee the 1.x polling cron had for free: it selected
executeAt <= now, so publishing early was structurally impossible.2.
server/services/action-service.js— makeremoveCronJoblog honestlystrapi.cron.removedoes not throw for an unknown task name, so the previous code loggedRemoved cron job for action Xon every instance, including ones that never had that job. During our incident investigation this log was actively misleading: it looked like the reschedule had been applied everywhere. The job list is now checked first (strapi.cron.jobs), and when nothing was removed locally the log says so and points at the possibility of a stale job elsewhere.Behaviour for existing users
No configuration changes, no schema changes, no API changes. In a single-instance deployment the new branch is never taken, so behaviour is identical. In a multi-instance deployment the difference is that an entry is no longer published ahead of its scheduled time.
Note that this makes stale jobs harmless but does not prevent them from being registered; that is what suggested fix 3 in #229 would address.
Testing
executeAtthat is later updated in the database, and confirmed the job now skips, logs the warning, and re-registers itself for the stored time.removeCronJoblog distinguishes the two cases.patch-packagepatch.Happy to adjust the tolerance value, log levels, or wording to your preference — and to fold in suggested fix 3 here instead if you would rather solve it at that level.