Skip to content

Agent-readable docs: Phase 2 — llms.txt and frontmatter descriptions#103

Merged
tbantle22 merged 1 commit into
devfrom
agent-docs-phase2
Jun 2, 2026
Merged

Agent-readable docs: Phase 2 — llms.txt and frontmatter descriptions#103
tbantle22 merged 1 commit into
devfrom
agent-docs-phase2

Conversation

@tbantle22
Copy link
Copy Markdown
Collaborator

Phase 2 of notes/agent-readable-docs.md, building on the Markdown alternates from Phase 1 (PR #101). Adds a hierarchical, agent-readable index of every docs page (llms.txt, per llmstxt.org) and starts wiring frontmatter description: into the entries.

What's in

File(s) Change
site/{dolt,doltlab,doltgres}/src/pages/llms.txt.ts (new) Static endpoint that emits llms.txt: H1 with the site title, blockquote summary, one H2 per top-level nav section, nested bullets for each page linking to the .md alternate. Descriptions come from the page's frontmatter description: field where present. Order matches nav.ts (so the sidebar).
site/{dolt,doltlab,doltgres}/_headers Explicit Content-Type: text/markdown for /docs/llms.txt and the bundle-root /llms.txt. Cloudflare otherwise ships text/plain.
site/{dolt,doltlab,doltgres}/package.json cp dist/docs/llms.txt dist/llms.txt in each build script, next to the existing _headers / _redirects / 404.html copies.
site/{dolt,doltlab,doltgres}/src/pages/404.astro Surface the llms.txt link as another discovery path from the helpful-404.
Frontmatter description: on a sample of high-traffic pages Dolt: introduction (what-is-dolt, installation, getting-started), the three top git concept pages (commits, branch, merge), products (hosted overview + cloning, dolthub overview), and the three big reference landings (CLI, dolt-sql-procedures, dolt-system-tables). DoltLab: what-is-doltlab + the two admin guides. Doltgres: introduction overview, installation, getting-started. Pages without a description still appear in llms.txt — they just don't get the descriptive suffix on the entry.
cypress/e2e/dolt/llms-txt.spec.ts (new) Verifies the file is served at /docs/llms.txt, starts with the title + blockquote summary, has one H2 per nav section, every entry links to a .md URL, and frontmatter descriptions appear after the colon.

Verification

  • Dolt llms.txt: 139 entries across 8 sections (167 lines). DoltLab: 97 lines. Doltgres: 77 lines.
  • Build emits the file at both dist/docs/llms.txt and dist/llms.txt for each site.
  • Cypress: llms-txt (4/4), markdown-alternate (4/4), navigation (5/5), dead-links (15/15), single-h1 (10/10) — all pass.

What's still ahead

Phase 3 (breadcrumbs, related-docs footer, broader description: backfill) per notes/agent-readable-docs.md. Phase 2 leaves us with a clean discovery surface even before that lands.

Phase 2 of notes/agent-readable-docs.md, building on the Markdown
alternates from Phase 1 (PR #101). Adds a hierarchical, agent-readable
index of every docs page and starts wiring frontmatter descriptions
into a sample of high-traffic pages.

- site/{dolt,doltlab,doltgres}/src/pages/llms.txt.ts (new)
  Static endpoint that emits llms.txt (https://llmstxt.org) — H1 with
  the site title, blockquote summary, one H2 per nav section, nested
  bullets for each item linking to the .md alternate. Descriptions
  come from the page's frontmatter `description:` field where present.

- site/{dolt,doltlab,doltgres}/_headers
  Explicit Content-Type: text/markdown for /docs/llms.txt and the
  bundle-root copy. Cloudflare otherwise ships text/plain.

- site/{dolt,doltlab,doltgres}/package.json
  cp dist/docs/llms.txt dist/llms.txt next to the existing _headers /
  _redirects / 404.html copies, so llms.txt is also served at the
  Cloudflare bundle root.

- site/{dolt,doltlab,doltgres}/src/pages/404.astro
  Surface the llms.txt link as another discovery path.

- Frontmatter descriptions on a sample of high-traffic pages
  Dolt: introduction (what-is-dolt, installation, getting-started),
  the three top git concept pages (commits, branch, merge), products
  (hosted overview + cloning, dolthub overview), and the three big
  reference landings (CLI, dolt-sql-procedures, dolt-system-tables).
  DoltLab: what-is-doltlab and the two admin guides (basic, enterprise).
  Doltgres: introduction overview, installation, getting-started.
  Pages without a description still appear in llms.txt — they just
  don't get the descriptive suffix on the entry.

- cypress/e2e/dolt/llms-txt.spec.ts (new)
  Verifies the file is served at /docs/llms.txt, starts with the title
  + summary, has one H2 per nav section, every entry links to a .md
  URL, and frontmatter descriptions appear after the colon.

Output: dolt llms.txt has 139 entries / 167 lines; doltlab 97 lines;
doltgres 77 lines. All three sites build clean; cypress llms-txt (4/4),
markdown-alternate (4/4), navigation (5/5), dead-links (15/15), and
single-h1 (10/10) all pass.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@tbantle22 tbantle22 merged commit 3a831f3 into dev Jun 2, 2026
7 checks passed
@tbantle22 tbantle22 deleted the agent-docs-phase2 branch June 2, 2026 18:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant