docs: add llms.txt ecosystem hub at site root

[timsaucer]

Which issue does this PR close?

• Addresses part of Make it easier for agents to generate datafusion-python code datafusion-python#1394 ("Make it easier for agents to generate datafusion-python code") — this is PR 6 in the implementation plan (comment): the upstream apache/datafusion llms.txt hub.

Rationale for this change

llms.txt is an emerging convention for exposing a machine-readable, agent-facing entry point at a site's docs root. Subprojects in the DataFusion ecosystem are starting to publish their own (apache/datafusion-python PR apache/datafusion-python#1505 added one). The main datafusion.apache.org site is the natural top-level discovery point for the whole ecosystem, so it should expose a hub llms.txt that points agents at:

• the core DataFusion (Rust) user / library / contributor guides and Rust API docs,
• each subproject's docs root, where agents following the llmstxt.org convention can probe <docs root>/llms.txt for project-specific guidance.

Net effect: an agent fetching https://datafusion.apache.org/llms.txt lands in a categorized directory of the entire ecosystem's agent guidance.

What changes are included in this PR?

• docs/source/llms.txt — new file, llmstxt.org schema. Sections: Core DataFusion (Rust), Subprojects, Optional. The Subprojects section links to docs roots (not pending llms.txt URLs) and includes a one-line note describing the probe convention so the hub stays correct as subprojects ship their own files.
• docs/source/conf.py — html_extra_path = ["llms.txt"] so Sphinx copies the file verbatim to the build output root, served at https://datafusion.apache.org/llms.txt.
• dev/release/rat_exclude_files.txt — exclude docs/source/llms.txt from the RAT license-header check (the file body is rendered markdown and cannot carry the standard .. comment header without breaking the format).

Are these changes tested?

No automated tests. The change is a single static file plus a Sphinx config line that mirrors a pattern already used in apache/datafusion-python (html_extra_path = ["llms.txt"], PR apache/datafusion-python#1505). Verification will be done at deploy time: confirm https://datafusion.apache.org/llms.txt resolves and renders.

Are there any user-facing changes?

Yes — adds a new public URL https://datafusion.apache.org/llms.txt. No existing pages are modified. No API changes.

[Copilot]

Pull request overview

Adds an agent-facing llms.txt “ecosystem hub” at the documentation site root (https://datafusion.apache.org/llms.txt) to improve automated discovery of DataFusion and subproject documentation.

Changes:

• Adds docs/source/llms.txt describing core DataFusion docs, subproject docs roots, and optional links.
• Configures Sphinx to copy llms.txt verbatim to the built site root via html_extra_path.
• Excludes docs/source/llms.txt from the Apache RAT license-header check.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.

File	Description
docs/source/llms.txt	New hub file linking to core DataFusion docs and subproject docs roots.
docs/source/conf.py	Copies llms.txt into the built HTML output root.
dev/release/rat_exclude_files.txt	Excludes the new llms.txt from RAT header enforcement.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.