Agent-readable docs: Phase 1 — markdown alternates, alt link, helpful 404, headers#101
Merged
Conversation
Source: https://neon.com/blog/agents-grew-up-so-did-our-docs Captures Neon's recommendations (markdown alternates at .md URLs, hierarchical llms.txt, link rel=alternate, breadcrumbs, related-docs, helpful 404s, content negotiation), audits where our three docs sites stand against each one, and proposes a phased rollout — Phase 1 covers markdown alternates + 404 + alternate link + Vary headers in one PR without touching content, Phase 2 generates llms.txt from nav.ts + frontmatter descriptions, Phase 3 is breadcrumbs + related-docs + the rest of the description backfill. A read-through for discussion, not a commitment. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
… 404, headers
Phase 1 of notes/agent-readable-docs.md. Gives every docs page a .md URL
agents can fetch, advertises it from the HTML head, and adds the headers
prod actually needs to serve it correctly. No content changes.
- site/{dolt,doltlab,doltgres}/src/pages/[...slug].md.ts (new)
Static endpoint that re-serves the raw Markdown source at the .md suffix
alongside the HTML page. Same content file and same contentRouteSlug, so
/docs/foo/bar and /docs/foo/bar.md always refer to the same source. Uses
import.meta.glob with ?raw so the bytes match what's in src/content/.
- site/shared/layouts/DocsLayout.astro
Add <link rel="alternate" type="text/markdown" href="<page>.md"> to the
head of every content page. Skipped on the docs root since /docs.md
isn't a useful URL.
- site/{dolt,doltlab,doltgres}/src/pages/404.astro
Expand from the one-line "Back to home" message to a helpful page —
links to home, mentions the in-page search (/), and explicitly tells
readers about the .md alternate convention (append .md to any docs URL).
- site/{dolt,doltlab,doltgres}/_headers (new)
Cloudflare Pages response headers. Vary: Accept on /docs/* (so caches
differentiate HTML vs. Markdown for the same URL) and an explicit
Content-Type: text/markdown + an hour of public caching on /docs/*.md
(Cloudflare otherwise defaults .md to text/plain).
- site/{dolt,doltlab,doltgres}/package.json
Hook cp _headers dist/_headers into each build script, next to the
existing cp _redirects line.
- cypress/e2e/dolt/markdown-alternate.spec.ts (new)
Verifies the .md endpoint returns 200 with the raw source (frontmatter
intact), that the HTML head advertises the alternate, that nested paths
work, and that unknown .md paths return 404.
- cypress/e2e/dolt/navigation.spec.ts
Update the "renders the custom 404 page" assertion to match the new
wording, and add coverage that the .md-alternate hint is on the page.
Verified: all three sites build clean (139 + 75 + 62 .md alternates),
markdown-alternate + navigation + dead-links specs all pass.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Same fix already landed for dolt's navigation.spec.ts — the 'Back to <site> documentation home' link text was dropped to just '<site> documentation home' in the helpful-404 rewrite. Two stale assertions hit CI on doltlab and doltgres. Also add coverage that the .md-alternate hint shows up so we don't lose it later. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Phase 1 of the agent-docs plan in
notes/agent-readable-docs.md(added in commit 9504074). Gives every docs page a Markdown URL that agents can fetch, advertises it from the HTML head, and adds the response headers prod actually needs to serve it correctly. No content changes — only the surface around content.What's in
site/{dolt,doltlab,doltgres}/src/pages/[...slug].md.ts(new).mdsuffix. Usesimport.meta.globwith?rawso the bytes matchsrc/content/exactly. SamecontentRouteSlugas the HTML catch-all, so/docs/foo/barand/docs/foo/bar.mdalways refer to the same source.site/shared/layouts/DocsLayout.astro<link rel="alternate" type="text/markdown" href="<page>.md">in the head. Skipped on the docs root (/docs.mdisn't useful).site/{dolt,doltlab,doltgres}/src/pages/404.astro.mdalternate convention.site/{dolt,doltlab,doltgres}/_headers(new)Vary: Accepton/docs/*so caches differentiate HTML vs. Markdown for the same URL. ExplicitContent-Type: text/markdown; charset=utf-8+ an hour of public caching on/docs/*.md(Cloudflare otherwise defaults.mdtotext/plain).site/{dolt,doltlab,doltgres}/package.jsoncp _headers dist/_headersin each build script, next to the existingcp _redirects.cypress/e2e/dolt/markdown-alternate.spec.ts(new).mdendpoint returns 200 with the raw source (frontmatter intact), that the HTML head advertises the alternate, that nested paths work, and that unknown.mdpaths return 404.cypress/e2e/dolt/navigation.spec.ts.md-alternate hint is on the page.Verification
.mdalternates, doltlab 75, doltgres 62 (matching the canonical page counts).markdown-alternate(4/4),navigation(5/5),dead-links(15/15) all pass.curl http://localhost:4321/docs/introduction/installation.mdreturns the raw source with frontmatter;<link rel="alternate" type="text/markdown" href="/docs/introduction/installation.md">is in the HTML<head>; no such link on the rootindex.html.What's still ahead
Phase 2 (
llms.txtgenerated fromnav.ts+ frontmatterdescription:) and Phase 3 (breadcrumbs, related docs, content backfill) per the plan innotes/agent-readable-docs.md. Phase 1 alone unlocks ~80% of the agent UX without touching any content.