Improve docs AI readiness assets

[gbarros]

Summary

• expand the LLM asset generator to emit canonical markdown indexes, llms-full.txt, section llms.txt files, sitemap.xml, robots.txt, an API catalog, and an agent skills index
• add homepage metadata, JSON-LD, and HTML discovery links for agent-readable resources
• ignore generated local public/ artifacts while preserving tracked source assets

Validation

• yarn build
• yarn lint (passes with existing warning in app/build/rpc/components/NodeAPIContent.tsx:144)
• JSON parse checks for out/.well-known/api-catalog and out/.well-known/agent-skills/index.json
• sitemap smoke check
• agent skill SHA-256 check

Notes

• This intentionally does not add MCP, OAuth/OIDC, WebMCP, commerce, or true HTTP Link-header support because this is a static docs site and those surfaces are out of scope for the repo.

I've ran it against a dev server to see the expected result:

[github-actions]

🚀 Preview Deployment

Your preview is ready: https://celestiaorg.github.io/docs-preview/pr-2513/

[Copilot]

Pull request overview

This PR expands the docs site’s “AI readiness” outputs by generating additional agent-consumable artifacts (LLM indexes, a full-text snapshot, sitemap/robots, and well-known discovery JSON) and by adding homepage metadata/discovery links so agents and crawlers can find those resources.

Changes:

• Extend scripts/generate-llms.js to emit canonical .md exports, llms-full.txt, section llms.txt files, sitemap.xml, robots.txt, and well-known discovery documents.
• Add site-level Next.js metadata + JSON-LD, and HTML discovery links for the generated agent resources.
• Update .gitignore to avoid committing generated artifacts under public/.

Reviewed changes

Copilot reviewed 2 out of 3 changed files in this pull request and generated 4 comments.

File	Description
scripts/generate-llms.js	Generates additional LLM/agent assets (markdown exports, indexes, sitemap/robots, well-known JSON).
app/layout.tsx	Adds metadata/JSON-LD and <link> discovery hints for agent-readable resources.
.gitignore	Ignores generated public/ artifacts while preserving tracked source assets (e.g., SKILL.md).

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[jcstein]

huge improvement from 0!

[jcstein]

LGTM !