fix(queues): reset deployment status to error when a job fails - #5078
fix(queues): reset deployment status to error when a job fails#5078NoiceHax wants to merge 2 commits into
Conversation
The deployment worker sets the service status to `running` before handing off to the deploy/rebuild helpers, but its catch block only logged the error. Any throw outside the helpers' own failure path — resolving the service, creating the deployment row, or an unreachable server while writing the error log — left applications, compose stacks and preview deployments pinned to `running` with no way to clear it from the UI. The worker now rolls the status back to `error`, guarding the rollback so a failing update cannot take the worker down with it. Closes Dokploy#4083
dosubot
Bot
added
the
size:S
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
| } | ||
| } catch (error) { | ||
| console.log("Error", error); | ||
| await markJobAsFailed(job.data); |
There was a problem hiding this comment.
When an application or compose deployment sets its status to done and the subsequent updateDeployment metadata write rejects, this catch calls markJobAsFailed and overwrites the resource status to error, causing the UI to report a successful deployment as failed.
Knowledge Base Used:
The rollback ran unconditionally, so a throw raised after the helper had already finished would relabel the result. deployApplication and deployCompose write the commit metadata in a finally block that runs after the status is set to done, and if that write rejects the queue caught it and overwrote done with error. markJobAsFailed now reads the current status first and only writes error while it is still running. A status the helper already settled is left alone.
|
Good catch on the post-success path. deployApplication and deployCompose write the commit metadata in a finally block that runs after the status is already set to done, so a reject there was reaching the queue catch and flipping a good deployment to error. Fixed in 38984ba. markJobAsFailed now reads the current status first and only writes error while it is still running, so a status the helper already settled is left alone. Added three cases for it (application done, compose done, helper-written error) plus one for a failing status lookup. 10 tests pass locally, biome and tsc --noEmit are clean. |
dosubot
Bot
added
size:M
1 participant
The deployment worker sets the service status to running before it calls the deploy or rebuild helpers, and the catch block only logged the error. The helpers set the status to error when they fail, but a throw outside that path, like resolving the service or an unreachable server while writing the error log, left applications, compose stacks and preview deployments stuck on running with no way to clear it from the UI.
The catch block now rolls the status back to error, but only while it is still running. A helper that already wrote done or error keeps its own result, so a late throw cannot relabel a deployment that actually succeeded. The rollback has its own try/catch so a failed write does not take the worker down.
Tests are in apps/dokploy/test/queues/deployments-queue.test.ts. They cover deploy, rebuild, compose and preview failures, and the done/error cases that should be left alone.
Closes #4083