upload zim to collection and create zimfarm task

[elfkuzco]

Rationale

This PR enhances the CMS by allowing editors to upload an already created zim from their local filesystem. The CMS on receiving such files creates the task on zimfarm which will advertise the book back to the CMS once checks and processing are done by a worker

Changes

• allow users to upload zims manually. uploads are done as S3 multipart uploads and API generates presigend URLs for uploading and executing the task. Presigned URLs for uploading expire after one day whereas the ones used by the worker expire after 7 days.
• add logic to authenticate with zimfarm API
• add task_id to book and corresponding UI link to view task on Zimfarm
• show tasks in collections views (where they were uploaded from)
• add shuttle task to delete zims of completed tasks
• add mill task to update status of requested tasks

This closes #322

[elfkuzco]

@benoit74 , if PR is approved, we will need to set policies on Wasabi to delete aborted/incomplete uploads and possibly retention policy on bucket too.

[elfkuzco]

Because uploads are done by editors, I have opted to keep the upload section in the collections page so collection_editor and global_editor can upload zims. But i'm stuck on the issue when book arrives and there's no title for it in which case collection_editor will not be able to act on it. Even worrying is that global editor when they try to promote book might attach book to a collection that is different from what the uploader intended. How do we go about this? Do we want to enhance promote book logic to only allow edits to the collection title when none is set? This way, when we determine book came from an uploaded zim, then, we fix it to that collection and path

@benoit74 , wdyt?

[codecov]

Codecov Report

❌ Patch coverage is 79.20354% with 94 lines in your changes missing coverage. Please review.
✅ Project coverage is 81.17%. Comparing base (380f7bd) to head (b5d3c43).

Files with missing lines	Patch %	Lines
backend/src/cms_backend/utils/s3.py	44.44%	21 Missing and 9 partials ⚠️
backend/src/cms_backend/api/token.py	40.00%	24 Missing and 3 partials ⚠️
backend/src/cms_backend/api/routes/collection.py	79.41%	11 Missing and 3 partials ⚠️
backend/src/cms_backend/utils/requests.py	42.85%	8 Missing ⚠️
backend/src/cms_backend/__init__.py	50.00%	2 Missing and 2 partials ⚠️
backend/src/cms_backend/db/book_actions.py	88.00%	1 Missing and 2 partials ⚠️
...src/cms_backend/mill/update_zimfarm_task_status.py	90.90%	2 Missing and 1 partial ⚠️
backend/src/cms_backend/db/requested_task.py	94.87%	2 Missing ⚠️
backend/src/cms_backend/api/context.py	94.44%	0 Missing and 1 partial ⚠️
backend/src/cms_backend/mill/context.py	0.00%	1 Missing ⚠️
... and 1 more

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #469      +/-   ##
==========================================
+ Coverage   81.13%   81.17%   +0.03%     
==========================================
  Files          64       69       +5     
  Lines        3881     4281     +400     
  Branches      473      507      +34     
==========================================
+ Hits         3149     3475     +326     
- Misses        581      643      +62     
- Partials      151      163      +12     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[benoit74]

Because uploads are done by editors, I have opted to keep the upload section in the collections page so collection_editor and global_editor can upload zims. But i'm stuck on the issue when book arrives and there's no title for it in which case collection_editor will not be able to act on it. Even worrying is that global editor when they try to promote book might attach book to a collection that is different from what the uploader intended. How do we go about this? Do we want to enhance promote book logic to only allow edits to the collection title when none is set? This way, when we determine book came from an uploaded zim, then, we fix it to that collection and path

Good point ; can you develop what you mean by "enhance promote book logic to only allow edits to the collection title when none is set"?

[elfkuzco]

Here's the UI when API sets collection titles in create_title action

[elfkuzco]

I realized we need the recipe_id in order to be able to reconstruct this path. So, I think maybe we shouldn't delete recipe till zimfarm has published book to CMS or mill has detected task has failed/canceled.

[kelson42]

I realized we need the recipe_id in order to be able to reconstruct this path. So, I think maybe we shouldn't delete recipe till zimfarm has published book to CMS or mill has detected task has failed/canceled.

sorry to intervene here as @benoit74 is out for a few days, but here there onviously no recipe-id to link to the book/title.

[elfkuzco]

sorry to intervene here as @benoit74 is out for a few days, but here there onviously no recipe-id to link to the book/title.

Zimfarm advertises the recipe_id as part of the payload that gets sent in the CMS notification. And before we create the task on zimfarm, we create a recipe and store the recipe_id. But, I'm proposing to delay the deletion of the recipe on zimfarm (which is currently done immediately after creation) till it gets advertised to CMS.