[Ulmo] Library v2 commit endpoint timeout with large Meilisearch indexes

[dwong2708]

Problem

The library commit endpoint /api/libraries/v2/<lib_key>/commit/ returns a 500 error when publishing 3+ blocks on instances with large Meilisearch indexes (500K+ documents). The publish itself succeeds (data is committed in Learning Core), but the post-publish search index update exceeds the hardcoded 15-second timeout.

This leaves the library MFE showing blocks in an intermediate state — marked as unpublished with no publish option visible — because the frontend relies on Meilisearch (/multi-search) for the component listing and the search index was never updated from the frontend's perspective.

Root Cause

wait_for_post_publish_events in tasks.py dispatches a Celery task and blocks with:

result.get(timeout=15, disable_sync_subtasks=False)

The Celery task fires LIBRARY_BLOCK_PUBLISHED for each block, which triggers library_block_published_handler -> upsert_library_block_index_doc.apply(). The .apply() call runs synchronously and includes _wait_for_meili_task() — a polling loop that blocks until Meilisearch confirms indexing.

On a 500K-document index, each document takes 5-7 seconds to index (inherent to Meilisearch's inverted index rebuild for large indexes). This is not resource-bound — we tested with 30Gi RAM / 6 CPU and observed the same 7s per document. Three blocks = 21 seconds, exceeding the 15-second timeout.

Proposed Short-Term Fix

Make the timeout configurable: Replace the hardcoded timeout=15 in dispatch_and_wait with getattr(settings, 'CONTENT_LIBRARIES_PUBLISH_TIMEOUT', 15) so operators can tune it for their index size without patching.

Batch document submissions: In send_events_after_publish (or send_change_events_for_modified_entities on master), collect all documents from the publish operation and submit them to Meilisearch in a single update_documents call instead of one document at a time. Meilisearch processes a batch of N documents as a single indexing operation — the rebuild time is roughly the same whether it's 1 or 10 documents. This would reduce a 3-block commit from 3x7s = 21s down to ~7s total.

Proposed Long-Term Fix

There are two architectural options to eliminate this problem at the root:

Option A: Decouple the library component listing from Meilisearch

Currently, the library authoring MFE retrieves the list of components directly from Meilisearch (/multi-search). This creates a hard dependency: after any write operation, the API must wait for Meilisearch to be consistent before responding, otherwise the frontend shows stale data.

The fix: use Meilisearch only for search functionality (full-text search, filtering, facets). For the library component listing (the default view showing all components), fetch directly from the Learning Core relational database via a dedicated REST endpoint.

With this approach:

• The commit endpoint writes to Learning Core (the source of truth) and returns immediately.
• The indexing events are triggered asynchronously in the background.
• The frontend fetches the updated component list directly from the database — which already has the committed state — and renders it immediately.
• Meilisearch catches up in the background; search results become consistent within seconds without blocking the user.
• This eliminates the coupling between publish latency and search indexing entirely, while preserving Meilisearch's role for full-text search where eventual consistency (a few seconds delay) is acceptable.

Option B: Fully async indexing with per-component loading state in the frontend

Instead of blocking the API response until all indexing is complete, make publish a fire-and-forget operation:

• The commit endpoint triggers the indexing events asynchronously (no result.get() wait) and returns immediately with a 200.
• The frontend (library authoring MFE) shows a per-component loading/publishing state — each block transitions individually from "publishing" to "published" as its indexing completes.
• The frontend polls or subscribes (via WebSocket/SSE) to the indexing status to know when each component is done.

This moves the responsibility of handling eventual consistency to the frontend, which is better positioned to give users real-time feedback on the progress of bulk operations. The backend no longer needs to block a synchronous HTTP request waiting for an external service (Meilisearch) to finish.

Environment

edx-platform release/ulmo.3 (also affects master)
Meilisearch v1.8 with ~500K documents in tutor_studio_content index
Tutor 21.x k8s deployment

Steps to Reproduce

1. Have a Meilisearch instance with >200K documents indexed
2. Create a content library with 3+ components
3. Edit all components (create pending draft changes)
4. Click "Publish All" (calls /api/libraries/v2/<lib_key>/commit/)
5. Observe 500 error (on ulmo.3) or stale MFE state (on master where exception is caught)
6. The content IS published (visible in the DB) but the search index is not updated within the timeout window, leaving the MFE showing stale state.

Evidence:

[dwong2708]

I’d appreciate your input here @ormsbee

[rodmgwgu]

I think the Batch document submission as described in the proposed short-term fix would be a huge win.

But also I think reviewing the arch as described in option A for the proposed long-term fix is needed, although I don't know what was the rationale behind the original decision.

[ormsbee]

On a 500K-document index, each document takes 5-7 seconds to index (inherent to Meilisearch's inverted index rebuild for large indexes). This is not resource-bound — we tested with 30Gi RAM / 6 CPU and observed the same 7s per document. Three blocks = 21 seconds, exceeding the 15-second timeout.

This seems crazy. It's expected behavior for Meilisearch to take 5-7 seconds to add a single document? Are we changing something in the index configuration itself that would cause it to force a rebuild of the entire thing?

FYI @bradenmacdonald, @kdmccormick, @openedx/interest-performance

[bradenmacdonald]

The library commit endpoint /api/libraries/v2/<lib_key>/commit/ returns a 500 error when publishing 3+ blocks on instances with large Meilisearch indexes (500K+ documents). The publish itself succeeds (data is committed in Learning Core), but the post-publish search index update exceeds the hardcoded timeout in dispatch_and_wait.

OK, something weird is going on here.

First of all, dispatch_and_wait is explicitly designed not to throw 500 errors when the indexing takes longer than the timeout. The entire point of dispatch_and_wait is that if indexing takes longer than the timeout, the indexing will still get done in the background, and a 200 response will be returned to the user anyways, before the indexing actually completes.

What exactly is the 500 error you're seeing?

A minor second point is that the dispatch_and_wait timeout is 10 seconds, and has never been 15 (I checked the git history). So: which version of the code are you using? This timeout should have absolutely no effect on whether or not things end up getting indexed or not. It just affects the user experience, and whether or not the frontend knows when to refresh the results (once the 200 response is returned, the frontend will refresh, so we try but don't insist on waiting to send the 200 only after the index is updated).

On a 500K-document index, each document takes 5-7 seconds to index (inherent to Meilisearch's inverted index rebuild for large indexes). This is not resource-bound — we tested with 30Gi RAM / 6 CPU and observed the same 7s per document. Three blocks = 21 seconds, exceeding the 15-second timeout.

I agree that this sounds problematic. I will note that in the past we have had performance issues that looked like Meilisearch issues but turned out to be bugs on the edxapp side. Have you measured this using the pure Meilisearch API directly?

And again, the timeout should not cause the indexing to fail.

Meilisearch v1.8 with ~500K documents in tutor_studio_content index

Meilisearch has released a lot of improvements since version 1.8 came out on May 7, 2024. In particular, version 1.12 introduced a next-generation indexer that's supposed to be 4x faster. So the very first thing I would do is measure indexing speed directly (using plain Meilisearch APIs) on a recent version of Meilisearch.

[bradenmacdonald]

The fix: use Meilisearch only for search functionality (full-text search, filtering, facets). For the library component listing (the default view showing all components), fetch directly from the Learning Core relational database via a dedicated REST endpoint.

This is making a big assumption that library authors are mostly using "the default view" without any full-text search, filtering, or facets applied. I'm not sure that's true, especially for relatively large libraries. I would assume that it's common to have at least some basic filter or facet applied, like showing only components, or viewing the contents of a single collection. As soon as that's the case, your suggested Option A no longer really helps (because then the results would still be using Meilisearch), and it just makes the code more complicated by having two different retrieval paths.

Batch document submissions: In send_events_after_publish (or send_change_events_for_modified_entities on master), collect all documents from the publish operation and submit them to Meilisearch in a single update_documents call instead of one document at a time.

This would be nice, and I'd like to do it, but we're a bit architecturally constrained at the moment since libraries and search are separate apps which rely on events to keep in sync. Achieving this would either require replacing the LIBRARY_BLOCK_CREATED, LIBRARY_BLOCK_UPDATED, etc. events with a LIBRARY_BLOCKS_CHANGED batch event, or update the low-level ENTITIES_DRAFT_CHANGED event to include data about context (library ID / course ID) and opaque keys. Either way is a fairly significant breaking change of all the related events.

Option B: Fully async indexing with per-component loading state in the frontend

This approach I really like, and I don't think there's any technical reason not to do it. It just needs the investment to make it happen. Although I would say that it probably shouldn't be done as a one-off thing and should be approached thoughtfully so it becomes a pattern that we can use throughout other APIs and other parts of the CMS/LMS UI. (Instructor tasks, import/export, etc.)

[dwong2708]

The library commit endpoint /api/libraries/v2/<lib_key>/commit/ returns a 500 error when publishing 3+ blocks on instances with large Meilisearch indexes (500K+ documents). The publish itself succeeds (data is committed in Learning Core), but the post-publish search index update exceeds the hardcoded timeout in dispatch_and_wait.

OK, something weird is going on here.

First of all, dispatch_and_wait is explicitly designed not to throw 500 errors when the indexing takes longer than the timeout. The entire point of dispatch_and_wait is that if indexing takes longer than the timeout, the indexing will still get done in the background, and a 200 response will be returned to the user anyways, before the indexing actually completes.

What exactly is the 500 error you're seeing?

A minor second point is that the dispatch_and_wait timeout is 10 seconds, and has never been 15 (I checked the git history). So: which version of the code are you using? This timeout should have absolutely no effect on whether or not things end up getting indexed or not. It just affects the user experience, and whether or not the frontend knows when to refresh the results (once the 200 response is returned, the frontend will refresh, so we try but don't insist on waiting to send the 200 only after the index is updated).

Sorry, I mixed the code between master and the Ulmo release. All of this issue is happening on Ulmo. I've updated the ticket to reference the correct location where the error is thrown — tasks.py on release/ulmo. The except TimeoutError there doesn't catch celery.exceptions.TimeoutError, which is why it surfaces as a 500 (I exactly see this error: celery.exceptions.TimeoutError: The operation timed out).

I see that on master this was refactored into dispatch_and_wait which correctly catches CeleryTimeout and passes gracefully — so the 500 is indeed an Ulmo-only bug. I understand that the indexing itself is not affected and it will eventually complete in the background. That said, from a UX perspective, when the response returns before indexing finishes, the frontend refreshes and shows stale data from Meilisearch — blocks appear stuck in an intermediate state until things catch up. This can be confusing for users since it looks like the publish didn't work.

[dwong2708]

On a 500K-document index, each document takes 5-7 seconds to index (inherent to Meilisearch's inverted index rebuild for large indexes). This is not resource-bound — we tested with 30Gi RAM / 6 CPU and observed the same 7s per document. Three blocks = 21 seconds, exceeding the 15-second timeout.

I agree that this sounds problematic. I will note that in the past we have had performance issues that looked like Meilisearch issues but turned out to be bugs on the edxapp side. Have you measured this using the pure Meilisearch API directly?

And again, the timeout should not cause the indexing to fail.

Meilisearch v1.8 with ~500K documents in tutor_studio_content index

Meilisearch has released a lot of improvements since version 1.8 came out on May 7, 2024. In particular, version 1.12 introduced a next-generation indexer that's supposed to be 4x faster. So the very first thing I would do is measure indexing speed directly (using plain Meilisearch APIs) on a recent version of Meilisearch.

Good points — I'll measure directly via the Meilisearch API and also test with v1.12+ to isolate. Will report back with results.

[dwong2708]

Hello, I bring the results. I tested directly via the Meilisearch API (single document PUT to our tutor_studio_content index with ~500K documents, on v1.8):

{
    "uid": 60950,
    "indexUid": "tutor_studio_content",
    "status": "succeeded",
    "type": "documentAdditionOrUpdate",
    "canceledBy": null,
    "details": {
        "receivedDocuments": 1,
        "indexedDocuments": 1
    },
    "error": null,
    "duration": "PT8.302285333S",
    "enqueuedAt": "2026-08-18T19:46:00.618974855Z",
    "startedAt": "2026-08-18T19:46:00.621689551Z",
    "finishedAt": "2026-08-18T19:46:08.923974884Z"
}

That confirms it takes ~8.3 seconds for a single document on the Meilisearch side alone, with no edxapp code involved.

[dwong2708]

I also tested batching — submitting 5 documents in a single API call to the same index:

{
    "uid": 60952,
    "indexUid": "tutor_studio_content",
    "status": "succeeded",
    "type": "documentAdditionOrUpdate",
    "details": {
        "receivedDocuments": 5,
        "indexedDocuments": 5
    },
    "duration": "PT8.130113417S",
    "enqueuedAt": "2026-08-18T19:58:07.56261503Z",
    "startedAt": "2026-08-18T19:58:07.564130698Z",
    "finishedAt": "2026-08-18T19:58:15.694244115Z"
}

5 documents in a batch take essentially the same time (~8.1s) as a single document (~8.3s). This suggests that batching document submissions on the edxapp side would be a significant improvement — a 5-block publish would go from 5×8s = 40s (current serial approach) down to ~8s total.

[ormsbee]

Do you know if the insert performance degraded linearly over time, or if there was a sudden spike? Is it possible that it's hitting some memory limit?

[dwong2708]

Do you know if the insert performance degraded linearly over time, or if there was a sudden spike? Is it possible that it's hitting some memory limit?

We don't have historical data to say whether it degraded linearly over time — the only data we have is from the tests I performed. I ran more than 10 tests and the results are consistent: most of them are above 6 seconds of execution time per document.

Regarding memory limits: we tested extensively. The Meilisearch process itself peaks at ~116Mi RSS during indexing (measured via kubectl top). We tested with memory allocations ranging from 2Gi up to 30Gi — the indexing time remained the same (~8s per document) regardless. The pod has never been OOM-killed (zero restarts). Storage is gp3 with 3000 baseline IOPS, so disk I/O shouldn't be a constraint either.

Based on all this, it seems inherent to the index rebuild cost at this document count rather than hitting any resource limit. That said, we haven't tested with a newer Meilisearch version yet — that's our next step.