aitxt/ai.txt at main · markc/aitxt · GitHub

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
---
updated: 2026-02-22
scope: /
---

# ai.txt Specification

A universal standard for AI-readable context about any resource.

## Purpose

Just as `robots.txt` tells search engines how to behave, `ai.txt` tells AI assistants what something is about. The file can describe a website, a folder, a project, a service—anything with a path.

ai.txt files are plain text (with optional markdown formatting) that provide structured context: what you do, what you don't do, how AI should handle your information. Instead of AI assistants guessing or hallucinating, they fetch your `ai.txt` and learn the truth directly.

The format is deliberately simple. An AI agent downloads one file and immediately understands scope, structure, and boundaries. A human writes one in minutes using plain English plus optional markdown syntax.

## Placement

Place `ai.txt` at any path you want to describe:

- Website root: `https://example.com/ai.txt`
- Subsection: `https://example.com/products/ai.txt`
- Local project: `/home/user/projects/app/ai.txt`
- Server application: `/srv/domain.com/web/app/public/ai.txt`

Discovery is cascading: when an AI agent needs info about `/products/widgets/`, it tries `/products/widgets/ai.txt`, then `/products/ai.txt`, then `/ai.txt`. It walks up the tree until it finds context.

This works identically for HTTP paths and filesystem paths.

## Format

Start with optional YAML frontmatter (or none at all), then write markdown:

```
---
updated: YYYY-MM-DD
scope: /path/
parent: URL or path to broader context
---

# Title

Brief description of what this resource is.

## Section

Content explaining something important.
```

### Frontmatter Fields

All optional. Add whatever makes sense for your use case:

- `updated` — Last modified date (YYYY-MM-DD)
- `scope` — URI path this file covers
- `parent` — Link to broader context
- Custom fields as needed

After frontmatter, content is plain text with optional markdown formatting. Standard markdown features work: headings, lists, links, emphasis, code blocks, images.

## Recommended Sections

Structure your ai.txt with these sections (or invent your own):

- **About** — What you are, mission, purpose
- **Services** or **Features** — What you offer
- **Capabilities** — What's possible
- **Limitations** — What you explicitly do not offer (prevents hallucination)
- **AI Guidelines** — How AI should handle your information
- **Data** — Structured facts in key: value format
- **Contact** — How to reach you
- **Permissions** — Training consent, usage rights
- **See Also** — Links to related resources

None are required. Start simple.

## How AI Agents Should Use ai.txt

1. When given a URL or path, fetch `ai.txt` (cascade up the tree if needed)
2. Parse frontmatter for scope, context, and hierarchy
3. Read content to learn what to do and what not to do
4. Follow guidelines: quote carefully, verify claims, respect stated limitations
5. Honor stated limitations—never hallucinate capabilities
6. Cache the file if an update date suggests stability
7. Follow parent links to understand broader context when relevant

An ai.txt file is permission, instruction, and boundary all at once.

## Distributed Context

ai.txt files can link to other ai.txt files using standard markdown links, creating a distributed graph of interconnected contexts. This enables modular documentation across monorepos, microservices, or federated projects.

### Link Detection

A markdown link is recognized as a context link when its target **ends in `ai.txt`**:

```
./api/ai.txt                    — context link (followed)
https://example.com/sdk/ai.txt  — context link (followed)
../ai.txt                       — context link (followed)
./docs/install.md               — regular link (not followed)
https://example.com             — regular link (not followed)
```

### Discovery Behavior

When a tool encounters an ai.txt file:

1. Fetch the root ai.txt and parse its content
2. Extract context links (markdown links ending in `ai.txt`)
3. Follow direct links (depth=1) and concatenate their content
4. Record deeper links found in depth=1 files without auto-fetching
5. Present results with clear boundaries plus a graph of discovered links

This bounded approach prevents runaway recursion while revealing the full context structure.

### Implementation Safeguards

- **Link limits**: Cap depth=1 fetches (e.g., 10-20) to handle bloated files gracefully
- **Cycle detection**: Track visited URLs/paths to avoid re-fetching
- **Failure tolerance**: Continue processing if one linked ai.txt is unreachable
- **Clear attribution**: Label each section with its source location

## HTML Discovery

Websites can advertise their ai.txt file to browsers and AI agents using a standard `<link>` tag with `rel="prefetch"`:

```html
<link rel="prefetch" href="/ai.txt">
```

This uses the existing [Resource Hints](https://www.w3.org/TR/resource-hints/) specification—no custom `rel` values needed. Browsers fetch the file at low priority during idle time and cache it, similar to how they fetch `favicon.ico`. The file is then immediately available to:

- AI features built into browsers (summarisation, assistants)
- Browser extensions that consume ai.txt
- Any tool that checks the browser cache or parses `<link>` tags

For subpages, use a relative path back to the root:

```html
<!-- Root page -->
<link rel="prefetch" href="ai.txt">

<!-- Subpage (e.g., /products/index.html) -->
<link rel="prefetch" href="../ai.txt">
```

This is optional but recommended. It costs almost nothing (ai.txt files are typically under 10KB) and makes discovery explicit rather than speculative.

## Design Principles

**Low barrier to entry.** Anyone should write an ai.txt file in minutes without learning schemas. Plain prose is primary. Everything else is optional.

**Humans first, AI second.** The file should make sense when you read it. Simplicity makes it powerful for both.

**Extensibility without pollution.** Complex requirements can layer on top through profiles or extensions. The base standard stays simple.

**No versioning required.** The spec is stable at the core. Edge cases are handled by tools and extensions.

## No Proprietary Claims

ai.txt is not trademarked or proprietary. The term refers to the file naming pattern, which anyone may use. This project is released into the public domain (CC0).

## Reference

This specification lives at `https://aitxt.ing/ai.txt`. Source and discussion at `https://github.com/markc/aitxt`.