From a2f431dea1bc8056d9651c8e1d2c88df9258ff40 Mon Sep 17 00:00:00 2001 From: Ali Serag Date: Thu, 23 Apr 2026 17:19:57 -0500 Subject: [PATCH] Adopt Content Signals Policy in robots.txt MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Adds an explicit Content Signals declaration (contentsignals.org, 2025) to the existing minimal wildcard file: Content-Signal: search=yes, ai-input=yes, ai-train=yes The Content Signals Policy — rolled out by Cloudflare to 3.8M+ domains since Sept 2025 — is the emerging post-user-agent-allowlist mechanism for expressing AI policy in robots.txt. Three signals: search — appear in traditional search indexes ai-input — used in live retrieval / AI answer citations (ChatGPT, Claude, Perplexity, etc.) ai-train — used to train or fine-tune AI models Flow developer docs are open source and we want maximum discoverability for humans and AI agents, so all three are set to yes. Rest of the file unchanged: still a wildcard allow-all (matches the pattern used by Mintlify, Supabase, Vercel, Anthropic, OpenAI, Stripe, Python docs, and developers.google.com). Also documents the companion /llms.txt and /llms-full.txt AI-indexed documentation paths plus the cadence-lang.org companion index. --- static/robots.txt | 17 ++++++++++++++--- 1 file changed, 14 insertions(+), 3 deletions(-) diff --git a/static/robots.txt b/static/robots.txt index 42bd7e932d..94ecf0d461 100644 --- a/static/robots.txt +++ b/static/robots.txt @@ -1,8 +1,19 @@ +# developers.flow.com robots.txt # algolia: 98E1096D4FD67E70 +# +# Content Signals Policy (contentsignals.org, 2025) — explicitly permit +# search indexing, live AI retrieval (ChatGPT / Claude / Perplexity / etc. +# citation), and AI training on all content. Flow developer docs are +# open source and intended to be maximally discoverable by humans and +# AI agents. -# Allow all bots User-agent: * -Disallow: +# Content-Signal: search=yes, ai-input=yes, ai-train=yes +Allow: / -# Sitemap Sitemap: https://developers.flow.com/sitemap.xml + +# Machine-readable AI indexes (complementary, non-standard): +# Curated index: https://developers.flow.com/llms.txt +# Full corpus: https://developers.flow.com/llms-full.txt +# Companion site: https://cadence-lang.org/llms.txt