Skip to content

Commit cf974d0

Browse files
mtoy-googly-mooglymtoy-googly-moogly
authored
Add CONTEXT.md and publish LLM workflow blog post (#272)
- Add CONTEXT.md with LLM reference documentation for the repository - Add "Integrating LLMs in the Malloy Development Workflow" blog post to index
1 parent b2dc385 commit cf974d0

2 files changed

Lines changed: 298 additions & 1 deletion

File tree

‎CONTEXT.md‎

Lines changed: 289 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,289 @@
1+
# CONTEXT.md - LLM Reference for Malloy Documentation Repository
2+
3+
This file contains essential context for LLMs working with this codebase that is not covered in existing documentation files.
4+
5+
## Project Overview
6+
7+
This is the **documentation repository** for [Malloy](https://github.com/malloydata/malloy), a semantic data modeling and query language. It is also the source for the Malloy blog.
8+
9+
- **Live site:** https://docs.malloydata.dev
10+
- **Repository:** https://github.com/malloydata/malloydata.github.io
11+
- **Build system:** Entirely custom TypeScript-based static site generator (not Jekyll, Hugo, etc.)
12+
13+
## The .malloynb File Format (CRITICAL)
14+
15+
The `.malloynb` extension stands for "Malloy Notebook" and uses a **custom format** parsed by `MalloySQLParser`. This is NOT standard markdown with fenced code blocks.
16+
17+
### Syntax
18+
19+
Content is separated by `>>>` delimiters:
20+
21+
```
22+
>>>markdown
23+
# Heading
24+
25+
Regular markdown content here.
26+
27+
>>>malloy
28+
source: flights is duckdb.table('flights.parquet')
29+
30+
>>>markdown
31+
More markdown content.
32+
33+
>>>malloy
34+
run: flights -> { aggregate: count() }
35+
```
36+
37+
### Key Points
38+
39+
1. **`>>>markdown`** - Markdown content block
40+
2. **`>>>malloy`** - Malloy code block (will be executed during build)
41+
3. **`>>>sql`** - SQL code block
42+
4. The parser treats these as "statements" in a notebook, not fenced code blocks
43+
5. Within markdown blocks, standard fenced code blocks (triple backticks) can be used for non-executable code examples
44+
6. YAML frontmatter at the very beginning is optional: `---\nlayout: documentation\n---`
45+
46+
### Code Execution
47+
48+
All Malloy code blocks are **executed at build time** using DuckDB. The query results are embedded in the generated HTML. This means:
49+
- All Malloy code must be valid and runnable
50+
- Sample data must exist for queries to work
51+
- Build failures will occur if Malloy code has errors
52+
53+
## Documentation Directives
54+
55+
See README.md for the full list of `#(docs)` options. Key additional detail:
56+
57+
- **`#(docs)`** (single `#`) - Query-level directives, applies to the query in this block
58+
- **`##(docs)`** (double `#`) - Model-level directives, applies to the model/source definition
59+
60+
## Build System Architecture
61+
62+
### Entry Point & Scripts
63+
64+
- **Main build:** `scripts/index.ts` (run via `tsx`)
65+
- **Document rendering:** `scripts/render_document.ts`
66+
- **Code execution:** `scripts/run_code.ts` (uses DuckDB)
67+
- **Syntax highlighting:** `scripts/highlighter.ts` (uses Shiki)
68+
- **Page structure/TOC:** `scripts/page.ts`
69+
70+
### Build Process
71+
72+
1. Register Handlebars partials from `/includes/`
73+
2. Load layouts from `/layouts/`
74+
3. Parse `table_of_contents.json` and `blog_posts.json`
75+
4. For each `.malloynb` file:
76+
- Parse with `MalloySQLParser`
77+
- Execute all Malloy code blocks via DuckDB
78+
- Render results as HTML
79+
- Apply Handlebars template
80+
- Validate links
81+
5. Generate search index (`js/generated/search_segments.js`)
82+
6. Copy Malloy render library to output
83+
84+
### Watch Mode
85+
86+
The system tracks dependencies between documents and model files. Changing `flights.malloy` only rebuilds documents that import it (tracked via `DEPENDENCIES` map in `run_code.ts`).
87+
88+
## Sample Data & Models
89+
90+
### Location
91+
92+
- **Malloy models:** `/models/` directory
93+
- **Parquet data files:** `/src/documentation/data/`
94+
95+
### Primary Models
96+
97+
| Model | Description |
98+
|-------|-------------|
99+
| `flights.malloy` | FAA flight data - primary example model |
100+
| `airports.malloy` | Airport reference data |
101+
| `carriers.malloy` | Airline carrier data |
102+
| `ecommerce.malloy` | E-commerce example |
103+
| `aircraft.malloy` | Aircraft details |
104+
105+
### Styles Files
106+
107+
`.styles.json` files alongside models define visualization defaults (e.g., `flights.styles.json`).
108+
109+
## Template System
110+
111+
### Handlebars Templates
112+
113+
- **Layouts:** `/layouts/` - page templates (`documentation.html`, `blog.html`, etc.)
114+
- **Partials:** `/includes/` - reusable fragments (`banner.html`, `ga.html`)
115+
116+
### Available Context Variables
117+
118+
In templates and frontmatter:
119+
120+
| Variable | Description |
121+
|----------|-------------|
122+
| `{{ site.baseurl }}` | Base URL for the site |
123+
| `{{ site.malloyRenderVersion }}` | Version of malloy-render library |
124+
| `{{ page.title }}` | Page title |
125+
| `{{ page.content }}` | Rendered page content |
126+
| `{{ page.url }}` | Page URL path |
127+
| `{{ page.footer }}` | Footer navigation HTML |
128+
129+
## Link Validation Rules
130+
131+
The build system validates all links. Key rules:
132+
133+
1. **Markdown links** (`[text](url)`) to internal docs MUST end with `.malloynb`
134+
- Correct: `[Sources](./source.malloynb)`
135+
- Wrong: `[Sources](./source)` or `[Sources](./source.html)`
136+
137+
2. **HTML links** (`<a href="">`) to internal docs MUST NOT include file extension
138+
- Correct: `<a href="/documentation/language/source">`
139+
- Wrong: `<a href="/documentation/language/source.html">`
140+
141+
3. **Hash anchors** are validated against actual heading IDs
142+
4. **External links** (http/https) are not validated
143+
5. **Absolute internal links** (starting with `/`) are discouraged except for `/slack`
144+
145+
## Blog Conventions
146+
147+
### Directory Structure
148+
149+
```
150+
src/blog/
151+
YYYY-MM-DD-slug/
152+
index.malloynb # Blog post content
153+
image.png # Optional images
154+
...
155+
```
156+
157+
### Metadata
158+
159+
Blog posts must be registered in `/src/blog_posts.json`:
160+
161+
```json
162+
{
163+
"title": "Post Title",
164+
"path": "/YYYY-MM-DD-slug",
165+
"author": "Author Name",
166+
"published": "YYYY-MM-DD",
167+
"subtitle": "Optional subtitle",
168+
"previewImage": "optional-preview.png"
169+
}
170+
```
171+
172+
Posts are automatically sorted by date (newest first). Navigation links (next/previous) are auto-generated.
173+
174+
## Table of Contents
175+
176+
Documentation navigation is defined in `/src/table_of_contents.json`:
177+
178+
```json
179+
{
180+
"contents": [
181+
{
182+
"title": "Section Name",
183+
"items": [
184+
{ "title": "Page Title", "link": "/path/to/page.malloynb" },
185+
{ "title": "Nested Section", "items": [...] }
186+
]
187+
}
188+
]
189+
}
190+
```
191+
192+
The TOC generates:
193+
- Sidebar navigation
194+
- Footer prev/next links
195+
- Page titles
196+
197+
## CI/CD Pipeline
198+
199+
See README.md for deploy instructions. Workflow details:
200+
201+
- **`.github/workflows/test.yaml`** - Runs `build-prod` on all PRs (must pass to merge)
202+
- **`.github/workflows/publish.yaml`** - On push to `main`, builds and force-pushes to `docs-release` branch
203+
- The `docs-release` branch contains only the built `/docs/` directory + `CNAME` file
204+
205+
## Style Conventions
206+
207+
See README.md for basic style rules. Additional details:
208+
209+
- **Meta-variables** in documentation use `<<variable>>` syntax (highlighted specially by the build system)
210+
- Code blocks should NOT have leading/trailing blank lines (build will warn)
211+
212+
## Key Dependencies
213+
214+
| Package | Purpose |
215+
|---------|---------|
216+
| `@malloydata/malloy` | Core Malloy compiler |
217+
| `@malloydata/render` | Result rendering (HTML tables, charts) |
218+
| `@malloydata/db-duckdb` | DuckDB connection for code execution |
219+
| `@malloydata/malloy-sql` | MalloySQL parser (parses .malloynb files) |
220+
| `@malloydata/syntax-highlight` | TextMate grammar for Malloy |
221+
| `shiki` | Syntax highlighting engine |
222+
| `handlebars` | Template engine |
223+
| `remark-*` | Markdown parsing |
224+
225+
## Cell Numbering
226+
227+
The build system tracks "cells" for linking to the GitHub web editor:
228+
229+
- Markdown blocks and Malloy blocks each count as cells
230+
- Cell numbering starts at 1
231+
- Links like `#C3` point to specific cells
232+
- Used for "Open in VSCode" links in rendered docs
233+
234+
## Common Gotchas
235+
236+
1. **All Malloy code runs at build time** - errors in Malloy code break the build
237+
2. **Link validation is strict** - builds fail on broken internal links
238+
3. **The `.malloynb` format is NOT standard markdown** - use `>>>malloy` blocks, not fenced code blocks
239+
4. **`##(docs) hidden`** must be on a Malloy block, not markdown
240+
5. **DuckDB connections are per-directory** - relative paths in Malloy are relative to the doc's directory
241+
242+
## Malloy Package Updates
243+
244+
To update Malloy packages:
245+
246+
```bash
247+
npm run malloy-update # Latest stable
248+
npm run malloy-update-next # Next/preview version
249+
```
250+
251+
For local development with linked Malloy packages:
252+
253+
```bash
254+
npm run malloy-link # Link local @malloydata packages
255+
npm run malloy-unlink # Unlink and reinstall from npm
256+
```
257+
258+
## Output Structure
259+
260+
Generated output goes to `/docs/` (gitignored):
261+
262+
```
263+
docs/
264+
documentation/
265+
language/
266+
query.html # From query.malloynb
267+
...
268+
blog/
269+
YYYY-MM-DD-slug/
270+
index.html
271+
index.html # Blog index
272+
css/
273+
js/
274+
generated/
275+
search_segments.js
276+
malloy-render-X.X.X.js
277+
img/
278+
CNAME # Added during deploy
279+
```
280+
281+
## File Type Summary
282+
283+
| Extension | Description |
284+
|-----------|-------------|
285+
| `.malloynb` | Malloy Notebook - documentation source |
286+
| `.malloy` | Malloy model file |
287+
| `.styles.json` | Visualization style definitions |
288+
| `.parquet` | Sample data files |
289+
| `.html` | Layout/partial templates or static HTML |

‎src/blog_posts.json‎

Lines changed: 9 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,13 @@
11
[
2-
{
2+
{
3+
"title": "Integrating LLMs in the Malloy Development Workflow",
4+
"path": "/2026-01-13-context-md-convention",
5+
"subtitle": "The CONTEXT.md Convention",
6+
"author": "Michael Toy",
7+
"published": "2026-01-13",
8+
"previewImage": "context-header.jpeg"
9+
},
10+
{
311
"title": "Metadata in Malloy",
412
"path": "/2025-06-16-annotations-and-tags",
513
"subtitle": "Annotations and Tags",

0 commit comments

Comments
 (0)