[Repo Assist] Add llms.txt/llms-full.txt generation

AGENTS.md

@@ -46,3 +46,15 @@ dotnet restore FSharp.Formatting.sln
 dotnet build FSharp.Formatting.sln --configuration Release
 dotnet test FSharp.Formatting.sln --configuration Release --no-build
 ```
+
+## Testing Locally Against Another Project
+
+After building the repo with `dotnet build`, run the tool directly from the build output in your project's directory:
+
+```bash
+# macOS / Linux
+/path/to/FSharp.Formatting/src/fsdocs-tool/bin/Debug/net10.0/fsdocs build
+
+# Windows
+\path\to\FSharp.Formatting\src\fsdocs-tool\bin\Debug\net10.0\fsdocs.exe build
+```

RELEASE_NOTES.md

@@ -3,6 +3,10 @@
 ## [Unreleased]
 
 ### Added
+* Generate `llms.txt` and `llms-full.txt` for LLM consumption by default; opt out via `<FsDocsGenerateLlmsTxt>false</FsDocsGenerateLlmsTxt>` in your project file. [#951](https://github.com/fsprojects/FSharp.Formatting/issues/951)
+* `llms-full.txt` now uses heading-per-entry format (`### [title](url)`) for better navigation structure. [#980](https://github.com/fsprojects/FSharp.Formatting/pull/980)
+* `llms.txt` index omits per-member API entries (individual properties/methods) to reduce size; `llms-full.txt` retains full member detail. [#980](https://github.com/fsprojects/FSharp.Formatting/pull/980)
+* `llms-full.txt` content now has HTML entities decoded and `--eval` warning lines stripped for cleaner LLM consumption. [#980](https://github.com/fsprojects/FSharp.Formatting/pull/980)
 * Add `<FsDocsAllowExecutableProject>true</FsDocsAllowExecutableProject>` project file setting to include executable projects (OutputType=Exe/WinExe) in API documentation generation. [#918](https://github.com/fsprojects/FSharp.Formatting/issues/918)
 * Add `{{fsdocs-logo-alt}}` substitution (configurable via `<FsDocsLogoAlt>` MSBuild property, defaults to `Logo`) for accessible alt text on the header logo image. [#626](https://github.com/fsprojects/FSharp.Formatting/issues/626)
 

docs/styling.md

@@ -76,6 +76,21 @@ For example:
 </PropertyGroup>
 ```
 
+## LLM-Friendly Output
+
+By default, `fsdocs build` generates `llms.txt` and `llms-full.txt` in the output root,
+following the [llmstxt.org](https://llmstxt.org/) convention. These files provide a structured
+index (and full content) of all documentation pages and API reference entries, making it easy
+to add documentation context to LLMs and AI coding assistants.
+
+To opt out, set the following property in your project file or `Directory.Build.props`:
+
+```xml
+<PropertyGroup>
+    <FsDocsGenerateLlmsTxt>false</FsDocsGenerateLlmsTxt>
+</PropertyGroup>
+```
+
 As an example, here is [a page with alternative styling](templates/leftside/styling.html).
 
 ## Customizing via CSS

src/fsdocs-tool/BuildCommand.fs

@@ -1287,6 +1287,80 @@
 
         startWebServerAsync serverConfig app |> snd |> Async.Start
 
+/// Helpers for generating llms.txt and llms-full.txt content.
+module internal LlmsTxt =
+
+    /// Decode HTML entities (e.g. " → ", > → >) in a string.
+    let private decodeHtml (s: string) = System.Net.WebUtility.HtmlDecode(s)
+
+    /// Strip FSharp.Formatting --eval warning lines from content.
+    let private stripEvalWarnings (s: string) =
+        s.Split('\n')
+        |> Array.filter (fun line ->
+            not (line.TrimStart().StartsWith("Warning: Output, it-value and value references require --eval")))
+        |> String.concat "\n"
+
+    /// Collapse three or more consecutive newlines into at most two.
+    let private collapseBlankLines (s: string) =
+        System.Text.RegularExpressions.Regex.Replace(s, @"\n{3,}", "\n\n")
+
+    /// Normalise a title: trim and collapse internal whitespace/newlines to a single space.
+    let private normaliseTitle (s: string) =
+        System.Text.RegularExpressions.Regex.Replace(s.Trim(), @"\s+", " ")
+
+    /// Decode HTML entities and remove --eval noise from content.
+    let private cleanContent (s: string) =
+        s
+        |> decodeHtml
+        |> stripEvalWarnings
+        |> collapseBlankLines
+        |> fun t -> t.Trim()
+
+    /// Build a section of llms.txt from a set of search index entries.
+    /// When <c>withContent</c> is true, entry content is appended under a heading per entry.
+    /// When false, entries are listed as bullet-point links (index format).
+    let buildSection sectionTitle (entries: ApiDocsSearchIndexEntry array) withContent =
+        if entries.Length = 0 then
+            ""
+        else
+            let sb = System.Text.StringBuilder()
+            sb.Append(sprintf "## %s\n\n" sectionTitle) |> ignore
+
+            for e in entries do
+                let title = normaliseTitle e.title
+
+                if withContent then
+                    sb.Append(sprintf "### [%s](%s)\n\n" title e.uri) |> ignore
+
+                    if not (System.String.IsNullOrWhiteSpace(e.content)) then
+                        sb.Append(cleanContent e.content) |> ignore
+                        sb.Append("\n\n") |> ignore
+                else
+                    sb.Append(sprintf "- [%s](%s)\n" title e.uri) |> ignore
+
+            sb.ToString()
+
+    /// Generate the text content of llms.txt (index) and llms-full.txt (with content).
+    /// Returns a tuple of (llms.txt content, llms-full.txt content).
+    let buildContent (collectionName: string) (entries: ApiDocsSearchIndexEntry array) =
+        let contentEntries = entries |> Array.filter (fun e -> e.``type`` = "content")
+        let apiEntries = entries |> Array.filter (fun e -> e.``type`` = "apiDocs")
+        // For the index, exclude per-member entries (identified by a '#' anchor in the URI).
+        let apiIndexEntries = apiEntries |> Array.filter (fun e -> not (e.uri.Contains("#")))
+        let header = sprintf "# %s\n\n" collectionName
+
+        let llmsTxt =
+            header
+            + buildSection "Docs" contentEntries false
+            + buildSection "API Reference" apiIndexEntries false
+
+        let llmsFullTxt =
+            header
+            + buildSection "Docs" contentEntries true
+            + buildSection "API Reference" apiEntries true
+
+        llmsTxt, llmsFullTxt
+
 type CoreBuildOptions(watch) =
 
     [<Option("input", Required = false, Default = "docs", HelpText = "Input directory of documentation content.")>]
@@ -1444,7 +1518,7 @@
         // See https://github.com/ionide/proj-info/issues/123
         let prevDotnetHostPath = Environment.GetEnvironmentVariable("DOTNET_HOST_PATH")
 
-        let (root, collectionName, crackedProjects, paths, docsSubstitutions), _key =
+        let (root, collectionName, crackedProjects, paths, docsSubstitutions, generateLlmsTxt), _key =
             let projects = Seq.toList this.projects
             let cacheFile = ".fsdocs/cache"
 
@@ -1675,6 +1749,13 @@
 
             File.WriteAllText(Path.Combine(rootOutputFolderAsGiven, "index.json"), indxTxt)
 
+        let generateLlmsTxt () =
+            if generateLlmsTxt then
+                let index = Array.append latestApiDocSearchIndexEntries latestDocContentSearchIndexEntries
+                let llmsTxt, llmsFullTxt = LlmsTxt.buildContent collectionName index
+                File.WriteAllText(Path.Combine(rootOutputFolderAsGiven, "llms.txt"), llmsTxt)
+                File.WriteAllText(Path.Combine(rootOutputFolderAsGiven, "llms-full.txt"), llmsFullTxt)
+
         /// get the hot reload script if running in watch mode
         let getLatestWatchScript () =
             if watch then
@@ -1949,6 +2030,7 @@
             // bespoke file for namespaces etc.
             let ok1 = ok1 && runDocContentPhase2 ()
             regenerateSearchIndex ()
+            generateLlmsTxt ()
             ok1 && ok2
 
         //-----------------------------------------
@@ -2010,7 +2092,8 @@
 
                             if runDocContentPhase1 () then
                                 if runDocContentPhase2 () then
-                                    regenerateSearchIndex ())
+                                    regenerateSearchIndex ()
+                                    generateLlmsTxt ())
 
                         Serve.refreshEvent.Trigger fileName
                     }
@@ -2031,7 +2114,8 @@
 
                             if runGeneratePhase1 () then
                                 if runGeneratePhase2 () then
-                                    regenerateSearchIndex ())
+                                    regenerateSearchIndex ()
+                                    generateLlmsTxt ())
 
                         Serve.refreshEvent.Trigger "full"
                     }

src/fsdocs-tool/ProjectCracker.fs

@@ -229,6 +229,7 @@ module Crack =
           FsDocsFaviconSource: string option
           FsDocsTheme: string option
           FsDocsWarnOnMissingDocs: bool
+          FsDocsGenerateLlmsTxt: bool
           FsDocsAllowExecutableProject: bool
           PackageProjectUrl: string option
           Authors: string option
@@ -262,6 +263,7 @@ module Crack =
               "FsDocsSourceFolder"
               "FsDocsSourceRepository"
               "FsDocsWarnOnMissingDocs"
+              "FsDocsGenerateLlmsTxt"
               "FsDocsAllowExecutableProject"
               "RepositoryType"
               "RepositoryBranch"
@@ -348,6 +350,7 @@ module Crack =
                   FsDocsFaviconSource = msbuildPropString "FsDocsFaviconSource"
                   FsDocsTheme = msbuildPropString "FsDocsTheme"
                   FsDocsWarnOnMissingDocs = msbuildPropBool "FsDocsWarnOnMissingDocs" |> Option.defaultValue false
+                  FsDocsGenerateLlmsTxt = msbuildPropBool "FsDocsGenerateLlmsTxt" |> Option.defaultValue true
                   FsDocsAllowExecutableProject =
                     msbuildPropBool "FsDocsAllowExecutableProject" |> Option.defaultValue false
                   UsesMarkdownComments = msbuildPropBool "UsesMarkdownComments" |> Option.defaultValue false
@@ -595,6 +598,7 @@ module Crack =
                 |> fallbackFromDirectoryProps "//RepositoryUrl"
               FsDocsTheme = projectInfos |> List.tryPick (fun info -> info.FsDocsTheme)
               FsDocsWarnOnMissingDocs = false
+              FsDocsGenerateLlmsTxt = projectInfos |> List.forall (fun i -> i.FsDocsGenerateLlmsTxt)
               FsDocsAllowExecutableProject = false
               PackageProjectUrl =
                 projectInfos
@@ -698,4 +702,4 @@ module Crack =
             |> List.choose (fun projectInfo -> projectInfo.TargetPath |> Option.map Path.GetDirectoryName)
 
         let docsParameters = parametersForProjectInfo projectInfoForDocs
-        root, collectionName, crackedProjects, paths, docsParameters
+        root, collectionName, crackedProjects, paths, docsParameters, projectInfoForDocs.FsDocsGenerateLlmsTxt

tests/FSharp.Literate.Tests/DocContentTests.fs

@@ -336,3 +336,173 @@ let ``ipynb notebook evaluates`` () =
 
     ipynbOut |> shouldContainText "10007"
 *)
+
+// --------------------------------------------------------------------------------------
+// Tests for LlmsTxt module (FsDocsGenerateLlmsTxt MSBuild property, on by default)
+// --------------------------------------------------------------------------------------
+
+open FSharp.Formatting.ApiDocs
+
+let makeEntry t title uri content =
+    { uri = uri
+      title = title
+      content = content
+      headings = []
+      ``type`` = t }
+
+[<Test>]
+let ``LlmsTxt buildContent produces correct header`` () =
+    let llmsTxt, llmsFullTxt = LlmsTxt.buildContent "MyProject" [||]
+    llmsTxt |> shouldContainText "# MyProject\n\n"
+    llmsFullTxt |> shouldContainText "# MyProject\n\n"
+
+[<Test>]
+let ``LlmsTxt buildContent with no entries produces header only`` () =
+    let llmsTxt, llmsFullTxt = LlmsTxt.buildContent "MyProject" [||]
+    llmsTxt |> shouldEqual "# MyProject\n\n"
+    llmsFullTxt |> shouldEqual "# MyProject\n\n"
+
+[<Test>]
+let ``LlmsTxt buildContent separates Docs and API Reference sections`` () =
+    let entries =
+        [| makeEntry "content" "Getting Started" "https://example.com/docs/getting-started" "Some intro text"
+           makeEntry "apiDocs" "MyModule.MyType" "https://example.com/reference/mytype" "Type docs" |]
+
+    let llmsTxt, _ = LlmsTxt.buildContent "MyProject" entries
+    llmsTxt |> shouldContainText "## Docs"
+    llmsTxt |> shouldContainText "## API Reference"
+
+    llmsTxt
+    |> shouldContainText "- [Getting Started](https://example.com/docs/getting-started)"
+
+    llmsTxt
+    |> shouldContainText "- [MyModule.MyType](https://example.com/reference/mytype)"
+
+[<Test>]
+let ``LlmsTxt llms.txt does not include content body`` () =
+    let entries =
+        [| makeEntry "content" "Getting Started" "https://example.com/docs/getting-started" "Detailed page content here" |]
+
+    let llmsTxt, _ = LlmsTxt.buildContent "MyProject" entries
+    llmsTxt |> shouldNotContainText "Detailed page content here"
+
+[<Test>]
+let ``LlmsTxt llms-full.txt includes content body`` () =
+    let entries =
+        [| makeEntry "content" "Getting Started" "https://example.com/docs/getting-started" "Detailed page content here" |]
+
+    let _, llmsFullTxt = LlmsTxt.buildContent "MyProject" entries
+    llmsFullTxt |> shouldContainText "Detailed page content here"
+
+[<Test>]
+let ``LlmsTxt llms-full.txt skips blank content`` () =
+    let entries = [| makeEntry "apiDocs" "MyModule" "https://example.com/reference/mymodule" "   " |]
+
+    let _, llmsFullTxt = LlmsTxt.buildContent "MyProject" entries
+    // Full file uses heading format per entry
+    llmsFullTxt
+    |> shouldContainText "### [MyModule](https://example.com/reference/mymodule)"
+    // Blank content should not produce extra blank lines beyond the heading line
+    llmsFullTxt.Contains("   ") |> shouldEqual false
+
+[<Test>]
+let ``LlmsTxt omits Docs section when no content entries exist`` () =
+    let entries = [| makeEntry "apiDocs" "MyModule" "https://example.com/reference/mymodule" "" |]
+    let llmsTxt, _ = LlmsTxt.buildContent "MyProject" entries
+    llmsTxt |> shouldNotContainText "## Docs"
+    llmsTxt |> shouldContainText "## API Reference"
+
+[<Test>]
+let ``LlmsTxt omits API Reference section when no apiDocs entries exist`` () =
+    let entries = [| makeEntry "content" "Guide" "https://example.com/docs/guide" "" |]
+    let llmsTxt, _ = LlmsTxt.buildContent "MyProject" entries
+    llmsTxt |> shouldContainText "## Docs"
+    llmsTxt |> shouldNotContainText "## API Reference"
+
+[<Test>]
+let ``LlmsTxt llms-full.txt uses heading format per entry`` () =
+    let entries = [| makeEntry "content" "Getting Started" "https://example.com/docs/getting-started" "Some content" |]
+
+    let _, llmsFullTxt = LlmsTxt.buildContent "MyProject" entries
+
+    llmsFullTxt
+    |> shouldContainText "### [Getting Started](https://example.com/docs/getting-started)"
+
+    llmsFullTxt |> shouldNotContainText "- [Getting Started]"
+
+[<Test>]
+let ``LlmsTxt llms-full.txt decodes HTML entities in content`` () =
+    let entries =
+        [| makeEntry
+               "content"
+               "Guide"
+               "https://example.com/docs/guide"
+               "use &quot;double quotes&quot; and &gt; greater-than" |]
+
+    let _, llmsFullTxt = LlmsTxt.buildContent "MyProject" entries
+    llmsFullTxt |> shouldContainText "use \"double quotes\" and > greater-than"
+    llmsFullTxt |> shouldNotContainText "&quot;"
+
+[<Test>]
+let ``LlmsTxt llms-full.txt strips eval warning lines from content`` () =
+    let content = "Some text\nWarning: Output, it-value and value references require --eval\nMore text"
+
+    let entries = [| makeEntry "content" "Guide" "https://example.com/docs/guide" content |]
+    let _, llmsFullTxt = LlmsTxt.buildContent "MyProject" entries
+    llmsFullTxt |> shouldNotContainText "--eval"
+    llmsFullTxt |> shouldContainText "Some text"
+    llmsFullTxt |> shouldContainText "More text"
+
+[<Test>]
+let ``LlmsTxt llms.txt excludes per-member API entries (URIs with hash)`` () =
+    let entries =
+        [| makeEntry "apiDocs" "MyModule" "https://example.com/reference/mymodule.html" "module docs"
+           makeEntry
+               "apiDocs"
+               "MyModule.myFunction"
+               "https://example.com/reference/mymodule.html#myFunction"
+               "member docs" |]
+
+    let llmsTxt, _ = LlmsTxt.buildContent "MyProject" entries
+
+    llmsTxt
+    |> shouldContainText "- [MyModule](https://example.com/reference/mymodule.html)"
+
+    llmsTxt |> shouldNotContainText "myFunction"
+
+[<Test>]
+let ``LlmsTxt llms-full.txt includes per-member API entries`` () =
+    let entries =
+        [| makeEntry "apiDocs" "MyModule" "https://example.com/reference/mymodule.html" "module docs"
+           makeEntry
+               "apiDocs"
+               "MyModule.myFunction"
+               "https://example.com/reference/mymodule.html#myFunction"
+               "member docs" |]
+
+    let _, llmsFullTxt = LlmsTxt.buildContent "MyProject" entries
+    llmsFullTxt |> shouldContainText "myFunction"
+
+[<Test>]
+let ``LlmsTxt normalises multi-line titles to single-line`` () =
+    let entries = [| makeEntry "content" "Fantomas\n" "https://example.com/docs/index.html" "Some content" |]
+
+    let llmsTxt, llmsFullTxt = LlmsTxt.buildContent "MyProject" entries
+    // Title must be on a single line — no embedded newline in the link text
+    llmsTxt |> shouldContainText "- [Fantomas](https://example.com/docs/index.html)"
+
+    llmsFullTxt
+    |> shouldContainText "### [Fantomas](https://example.com/docs/index.html)"
+
+    llmsTxt |> shouldNotContainText "Fantomas\n"
+
+[<Test>]
+let ``LlmsTxt collapses excessive blank lines in content`` () =
+    let content = "First paragraph\n\n\n\n\nSecond paragraph"
+
+    let entries = [| makeEntry "content" "Guide" "https://example.com/docs/guide" content |]
+    let _, llmsFullTxt = LlmsTxt.buildContent "MyProject" entries
+    // Should not contain 3 or more consecutive newlines
+    llmsFullTxt.Contains("\n\n\n") |> shouldEqual false
+    llmsFullTxt |> shouldContainText "First paragraph"
+    llmsFullTxt |> shouldContainText "Second paragraph"