feat(api): migrate from Celery to RQ and expose queue position#143
Merged
Conversation
Replace Celery with RQ to allow reporting pending task position and in-progress count in task status responses, resolving #107. Co-authored-by: Cursor <cursoragent@cursor.com>
github-actions
Bot
added
documentation
Avoid resolving the Redis host when loading the API Taskfile so CI tasks like project-check can run without a Redis instance. Co-authored-by: Cursor <cursoragent@cursor.com>
Add explicit enqueue settings typing for mypy and remove an unused import in the Redis helper so quality checks pass again. Co-authored-by: Cursor <cursoragent@cursor.com>
Branch coverage •
|
||||||||||||||||||||
…etup - Changed RQ-dashboard dependency from a Git URL to a PyPI source in multiple configuration files for consistency. - Enhanced devcontainer configuration by adding host mapping and forwarding ports for better development experience. - Updated Dockerfile to include docker-compose-plugin for improved container management. - Adjusted task scripts to reference the correct path for Redis wait script and added a new script for cleaning stale RQ workers. - Made database migration scripts more robust by checking for existing columns before adding or dropping them.
- Changed the parameter type of the `is_stale` function from `Worker` to `BaseWorker` to align with the updated RQ library structure.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Closes #107
Migrates the Ecoindex API task backend from Celery to RQ, which allows exposing queue status information that Celery does not provide.
When submitting a new analysis or polling a pending task, the API now returns:
queue_position— position in the waiting queue (0= next to run,nullwhen running or completed)tasks_in_progress— number of tasks currently being processed on the queueAPI changes
GET /v1/tasks/ecoindexes/{id}response includesqueue_positionandtasks_in_progresswhile the task is pendingInfrastructure & dev experience
ecoindexandecoindex_batchqueues)task start-rq-dashboardon port9181)start_rq_workers.sh(shared between Docker and local dev) andwait_redis.pyfor reliable Redis discovery in devRQ_FAILURE_TTL,RQ_JOB_TIMEOUT,RQ_RESULT_TTL,RQ_WORKERSTest plan
task start-devand verify API, worker, and RQ dashboard are runningPOST /v1/tasks/ecoindexes/and confirm a job ID is returnedGET /v1/tasks/ecoindexes/{id}while pending and verifyqueue_positionandtasks_in_progressare presentqueue_positionincreases for later tasksstatus: SUCCESSwithecoindex_resultRQ_WORKERS> 1Made with Cursor