[Repo Assist] Add llms.txt/llms-full.txt generation#980
[Repo Assist] Add llms.txt/llms-full.txt generation#980github-actions[bot] wants to merge 17 commits intomainfrom
Conversation
Implements support for generating llms.txt and llms-full.txt files alongside the documentation output when --generatellmstxt is passed to fsdocs build or fsdocs watch. - llms.txt: title/link index of all docs and API reference entries - llms-full.txt: same with full page content included Files follow the llmstxt.org convention, making project documentation easily consumable by LLMs and AI coding assistants. Closes #951 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
|
✅ Pull request created: #980 |
|
/repo-assist change the flag in this PR to |
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
|
✅ Commit pushed: |
|
🤖 This is an automated response from Repo Assist. Done! Renamed
|
|
/repo-assist We need some kind of testing added for this |
Extract llms.txt generation logic into LlmsTxt module and add 8 unit tests covering: header generation, empty entries, Docs/API Reference section separation, index-only vs full-content modes, blank content skipping, and conditional section omission. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
|
✅ Commit pushed: |
|
🤖 This is an automated response from Repo Assist. Done! Added testing for the
The
All 8 tests pass (
|
src/fsdocs-tool/BuildCommand.fs
Outdated
| [<Option("clean", Required = false, Default = false, HelpText = "Clean the output directory.")>] | ||
| member val clean = false with get, set | ||
|
|
||
| [<Option("llms", |
There was a problem hiding this comment.
I think it makes more sense to have this as fsproj/msbuild configuration rather than a new feature flag.
nojaf
left a comment
There was a problem hiding this comment.
Having to pass --llm on everything invocation doesn't seem the way to go for me.
I'd rather enable this once and then never forget about it.
Yup agreed. It's possible it should be on by default too? Gven use of LLMs is now normal. |
Seems like a reasonable default yes. |
|
/repo-assist update this PR taking this into account:
Please make sure all docs are updated. |
…enabled by default) Instead of requiring users to pass --llms on every fsdocs build/watch invocation, read the new <FsDocsGenerateLlmsTxt> MSBuild property from the project file or Directory.Build.props. The property defaults to true, so llms.txt and llms-full.txt are generated automatically. Projects that want to opt out can set <FsDocsGenerateLlmsTxt>false</FsDocsGenerateLlmsTxt>. Closes #951 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
|
✅ Commit pushed: |
|
🤖 This is an automated response from Repo Assist. Done! Replaced the What changed:
Behaviour:
The property can be set in the project file (
|
|
/repo-assist Make the default for the MSBuild property for llms text to be on by default please and adjust test and documentation for this. Update release notes |
… (on by default) The LlmsTxt tests were added when this was a CLI flag. Update the section comment to accurately describe the current implementation as an MSBuild property that is enabled by default. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
|
/repo-assist I'm still not entirely happy with the output here. Example Notice that are a lot of blank lines between sections. Improve the code and verify your own work by running
this will create files in |
… blank lines - Trim and collapse internal whitespace in entry titles so link text is always on a single line (e.g. 'Fantomas\n' → 'Fantomas') - Collapse 3+ consecutive newlines to at most 2 in cleaned content, preventing large gaps between sections in llms-full.txt - Trim leading/trailing whitespace from cleaned content - Add two new tests covering both behaviours Addresses feedback from @nojaf on PR #980. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
|
✅ Commit pushed: |
|
🤖 This is an automated response from Repo Assist. I've addressed your feedback from the latest review: What was fixed:
Two new tests covering both behaviours were added and all 15 LlmsTxt tests pass.
|
|
/repo-assist So, right now I'm not convinced this is useful for LLMs. If the current project has an You need to update this and I would argue that markdown output should also be generated by default when @dsyme do you agree? |
@nojaf I agree. Would be interesting to compare with other llms.txt for other langauges |
|
@nojaf Do you want FsDocsGenerateLlmsTxt on by default (so markdown will also be generated by default)? I think I would be OK with this given the modern era we're in |
|
/repo-assist please act on my comment in #980 (comment) |
|
@dsyme yes, enabled by default make a lot of sense. |
…enerateLlmsTxt is enabled - When FsDocsGenerateLlmsTxt is enabled (the default), prefer markdown templates for both API docs and doc content so that .md files are written alongside HTML output. - Bundle docs/_template.md as a package template so that markdown output is generated even when the user has not provided their own _template.md. - LlmsTxt.buildContent now accepts docContentUsesMarkdown and apiDocUsesMarkdown flags and transforms URIs accordingly: .html links become .md, and markdown API-doc URIs (InUrl='') get .md appended. - API docs template selection now prefers _template.md over _template.html when generateLlmsTxt is enabled. - DocContent.Convert accepts an optional defaultMdTemplate parameter that is passed as the initial markdown template to processFolder. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
|
Commit pushed:
|
This comment has been minimized.
This comment has been minimized.
|
/repo-assist I'm still a little torn about the Can you add some sort of warning message that if no Also fix the build! |
|
/repo-assist See comment from @nojaf above. Also about this:
This is indeed a problem - turning on FsDocsGenerateLlmsTxt should also generate markdown - it shouldn't switch to markdown. I didn't actually see in the docs anything about how to turn on markdown documentation generation and whether that can be done in addition to HTML or other output, so we probably have documentation gaps too. Do what you can but we might have to rethink this if it gets too complex |
|
@dsyme I believe it happens now if the user has a |
…stead of it - Restore HTML-first template ordering for API docs (was incorrectly switched to markdown-first when FsDocsGenerateLlmsTxt was enabled, causing API docs to be emitted as .md instead of .html when _template.md existed) - Remove markdown fallback when no user templates found; use HTML default instead - Add warning when FsDocsGenerateLlmsTxt is enabled but no _template.md found, advising users to add one for better LLM-friendly output - Fix StringComparison.Ordinal in StartsWith call (code scanning alert) - Fix test compilation: update LlmsTxt.buildContent calls to pass required docContentUsesMarkdown and apiDocUsesMarkdown arguments Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
|
Commit pushed:
|
|
🤖 This is an automated response from Repo Assist. I've pushed fixes addressing Changes made1. Fix: The template ordering for API docs has been restored to HTML-first, regardless of The doc content layer already generates 2. Add: Warning when no When 3. Fix: Build errors in tests The 4. Fix: Code scanning alert Added Note on "also generate markdown for API docs": Currently API docs can only be generated in one format per run (HTML or Markdown). Generating both in the same run would require a second pass through Test status: Build succeeds, all 116 literate tests pass.
|
|
@nojaf Not sure this feature is worth it if it only kicks in when generating markdown docs (and not HTML). wdyt should we ask it to do the work to allow two doc type generations? I guess that's the solution? |
|
@dsyme I would do the following: FsDocsGenerateLlmsTxt is on by default, and because it is on we generate markdown as well regardless of the presence of If you turn of FsDocsGenerateLlmsTxt and have no |
🤖 This is an automated PR from Repo Assist, an AI assistant for this repository.
Closes #951
Summary
Adds a
--generatellmstxtflag tofsdocs buildandfsdocs watch. When enabled, two files are written to the output root:llms.txt— a Markdown index with links to all documentation pages and API reference entries, following the [llmstxt.org]((llmstxt.org/redacted) conventionllms-full.txt— same as above but with full page content included after each entryThis makes it easy to add documentation context for F# projects to LLMs and AI coding assistants.
Usage
Trade-offs
llms-full.txtcan be large for projects with extensive API docs. Users can choose which file to expose.Test Status
dotnet build src/fsdocs-tool/fsdocs-tool.fsproj -c Release— succeeded, 0 errorsdotnet test tests/FSharp.ApiDocs.Tests/FSharp.ApiDocs.Tests.fsproj -c Release— 68 passed, 0 failed