Search index does not update when pages are added or removed

[mabijkerk]

Context

The site search does not reflect changes to the documentation. Deleted pages keep showing up in the search bar, and newly added pages are not searchable, even after the change is deployed.

Search runs on hosted Algolia DocSearch (the algolia block in docusaurus.config.js). The index lives on Algolia's servers and is filled by a crawler that runs on its own schedule, decoupled from our Netlify deploys. There is no crawler config or CI workflow in this repo, so nothing re-indexes the site when we deploy — the index only changes when Algolia's crawler happens to run.

Examples of both directions:

• Deleted page still indexed: docs/main/methane.md was deleted, but the record is still live in the index:

  objectID: 0-https://docs.energytransitionmodel.com/main/methane/
  lvl1:     "Methane"
  

• New page not indexed: docs/main/liquid-fuels.md was added on 2025-10-30 (~9 months ago) and is still not searchable — a search for liquid fuels returns the Hydrogen and Cost methods pages but never /main/liquid-fuels/.

Steps to reproduce

Version 2026-06 release tag:

1. Delete (or rename) a page, e.g. docs/main/methane.md, and deploy.
2. Open the docs site and search for the page title (e.g. methane).
3. The deleted page still appears as a result. The same happens in reverse: a newly added page does not appear until the next crawl.

Proposed solution

Multiple solutions are possible:

• Replace hosted Algolia with a local (offline) search plugin such as @easyops-cn/docusaurus-search-local, so the index is built during yarn build and always matches what is deployed.
• Keep Algolia but trigger a reindex on deploy (Netlify hook or GitHub Action calling the crawler), if we want to keep Algolia's search quality.

In determining the solution, we should keep in mind this additional issue with Algolia: #247.