docs: add plan for docs-driven skill generation

[Ethan-Arrowood]

Summary

Adds docs/plans/docs-driven-skills.md — the design for auto-generating Harper skill rules from the documentation repo so they stop drifting from the source of truth.

This PR is the plan document only. No implementation, no behavior change. Merging it commits the team to the approach; the work itself is broken into phases inside the doc.

Background

Today the 20 rule files under harper-best-practices/rules/ are maintained by hand — sometimes with agent assistance, but a human still has to notice a docs change, prompt the rewrite, and open a PR. Drift example: the rest: true config prereq existed in reference/rest/overview.md long before the skill was patched to mention it.

What the plan covers

• Concepts. Defines rule vs. skill vs. AGENTS.md vs. manifest, so the file purposes are explicit. Two generation modes: generate (auto-produced from docs) and synthesized (hand-authored, for content with no canonical docs source).
• Guiding principle. Humans own the rule taxonomy; automation owns keeping rule bodies in sync with their declared sources.
• User stories. Five workflows covering automated regen on docs prose changes, adding a new rule manually, authoring synthesized rules, fixing the manifest when docs structure changes, and adding a whole new skill.
• Phased migration.
  • Phase 0 — manifest + lightweight validator, every existing rule mapped as synthesized (no behavior change today)
  • Phase 1 — one rule end-to-end (vector-indexing)
  • Phase 2 — expand to obvious .md-only rules
  • Phase 3 — flat-markdown export in HarperFast/documentation (Docusaurus plugin that flattens MDX components to plain markdown alongside the HTML build)
  • Phase 4 — MDX-sourced rules + observability
  • Phase 5 — steady state
• Developer documentation. Substantially expands the existing .github/CONTRIBUTING.MD to explain repo anatomy, the generation pipeline, common contributor tasks, and what's automated vs. manual.
• Validation layer. Spec for validate-generated.mjs (manifest completeness, provenance comments, must-cover assertions, MDX leakage, cross-link integrity, AGENTS.md round-trip).
• Alternative: pointer strategy. Documented as a known fallback only — explicitly not in scope of this plan. If the generation approach disappoints, the team can revisit.

Reviewer notes

• The plan does not lock down implementation specifics that should be debated during Phase 0 (manifest YAML shape is illustrative, not final).
• Phase 3 is the largest cross-repo commitment and the one most worth a careful read. It assumes the docs team (same person, in this case) is willing to ship a flat-markdown export. If that's contentious, surface it now.
• Phase 3 can run in parallel with Phase 2 — Phase 2 candidates all source from .md files.
• The plan file itself is a planning artifact; once Phase 5 lands it can be archived. .github/CONTRIBUTING.MD is the long-lived companion.

🤖 Generated with Claude Code

[cb1kenobi]

Fascinating!

[kriszyp]

This is fantastic, this is an excellent approach, thank you for putting this together!
I left some comments to consider as you build this, but I certainly think you can move forward with.

[kriszyp]

This is the right approach for the beginning, but the goal would be to hopefully eliminate this, based on your guiding principle, right?

Anything that re-renders prose when docs change is automated

[Ethan-Arrowood]

You mean like AI reviewed (or just no review if we trust it enough?)

[kriszyp]

And likewise once we gain confidence in updates, hopefully additions would be fully automated (no human) as well per the guiding principle.

[kriszyp]

Just to clarify, the analysis here is that our current MDX files have too much noise from JSX components/tags, and that stripping it down to cleaner MD files? And that simply offering agent guidance for the renderer ("please ignore JSX components") is likely to be less efficient (for agents/LLMs) than reading docs with AST cleansing?
I don't know if this influences the technique, but I believe our source files are much closer to what we want agents to read than our generated HTML. A technique than can directly translate source to "flat" markdown without dealing with the HTML seems ideal.
I think this also solves the long-standing question of providing agent-optimized Markdown for public AI crawlers, hopefully in an efficient manner.

[Ethan-Arrowood]

Yes and I've started using MDX for reusing certain bits of info. See this example: https://github.com/HarperFast/documentation/blob/547bbfc679fd772f591c88d032be22ab5da67133/learn/getting-started/create-your-first-application.mdx?plain=1#L30

We don't do this in the reference materials yet, but we likely will.

While the <Tab> MDX element isn't much of an issue, the content ones in https://github.com/HarperFast/documentation/tree/main/src/components/learn do need to be like rendered for it to be properly included.

[kriszyp]

We are also going to consider regenerating existing skills content from the documentation source, right? I believe we should at least try that. Perhaps there are some existing skills (that would be considered "synthesized") that might be deemed high quality, but in general we want to actually replace our existing content in skills with the generated content (otherwise they are stuck in synthesized state hindering more automated regeneration).

[Ethan-Arrowood]

Yes - this is part of the plan somewhere. We start with everything synthesized, and then we migrate slowly once things start working.

[kriszyp]

We have already gone down the path of acquiring an Anthropic API key for PR reviews, so hopefully that can be followed for this. I believe the API key generation is easy, just making sure we have the secret setup.
It seems like it is also worth considering the use of Gemini, and maybe Claude can build an option to use either. Again, we have tons of credits, so if economics start to play into this, that could be helpful (although I suspect this should be relatively inexpensive).

[kriszyp]

Is the hypothesis that generated skills/rules should be more succinct and conducive to LLMs remaining attentive to reading them, rather than LLMs starting to "skim" long embedded/linked documentation? Or is it partly the cleansing of JSX that benefits the skill? A "release-asset" (of cleansed flat markdown) as the source of skills could address that. Perhaps we might also want to consider more flexibility/hybrid-ness and offer "synthesized", "generated", and "flat" (or "direct") with the third option indicating that the source (flat) markdown file should be imported as-is without any LLM summarization.
I will say that I do believe the hypothesis that LLM summarization is likely to be better. But these might be good options to retain and compare.

[Ethan-Arrowood]

I love this idea. Really this is just based on the brief conversation in Slack. @dawsontoth seems to prefer we generate rather than use direct. I don't have any reason to back one way or another. I specifically included the direct method as a backup incase we want to go that way. I'll have the plan incorporate it as a third option instead so we can have the best of both worlds immediately.

[dawsontoth]

The particular word I used was "transform", not generate.

[Ethan-Arrowood]

If its not direct, then what would be the difference between generate and transform?

[kriszyp]

These are three modes and the (expanded) alternate names to consider?

• synthesized/manual/human-crafted - Source of truth is in /skills
• generated/transformed - Source of truth in docs (llms.txt plugin output), but summarized by AI into skills.
• direct/flat/linked - Source of truth in docs (llms.txt plugin output), directly copied into skills
  I know this is merged, and I am flexible on naming.

[Ethan-Arrowood]

yeah this is just tooling so we can change it easily. I like what exists, but open to changing if others prefer some of the alternatives

[Ethan-Arrowood]

Update for previously-approving reviewers — second commit (4712e52) summary

This commit substantially revises the plan based on follow-up design discussion. The high-level goal and guiding principle are unchanged; the implementation strategy and structure are meaningfully different. Worth re-reviewing.

TL;DR: The plan grew formal schema sections, added a third generation mode (direct), switched Phase 3 from a custom MDX→MD pipeline to adopting an existing community plugin, reordered phases so plugin adoption comes first, and committed to an offline-first data flow (no network fetches at sync time).

What's new

• Manifest Schema section — formal field reference table for rules.manifest.yaml with a worked example. Defines every field (rule, description, category, priority, order, mode, sources[], must_cover, cross_links), which are required, and their types/semantics.
• Rule Frontmatter Schema section — formal field reference for the metadata block in each rules/*.md frontmatter. Makes the manifest↔frontmatter relationship explicit: manifest is the declaration; frontmatter is the last-generated snapshot.
• Generation Lifecycle section — step-by-step description of what a regen run does: manifest lint → per-rule resolve/hash/skip-check/produce-body → AGENTS.md refresh → validate → diff check → PR open.
• direct mode — third generation mode alongside generate and synthesized. Verbatim flat-markdown import with no LLM call, for cases where docs prose is already concise and agent-friendly. Per-rule mode is reversible at any time.
• Story 6 — new user story showing direct mode picked for the automatic-apis rule.

What changed in approach

• Phase 3 strategy — was "build a custom MDX→MD pipeline with per-component handlers"; now "adopt @signalwire/docusaurus-plugin-llms-txt" (MIT-licensed, ~19k weekly downloads, used by Cedar / MLflow / others; explicitly handles multi-instance docs via Docusaurus's postBuild route data). Backed by extensive evaluation summarized in facebook/docusaurus#10899. The custom-pipeline approach is preserved as a fallback in the same section.
• Phase order — plugin adoption moved from Phase 3 to Phase 1. Since all source resolution now depends on the plugin's output, single-rule end-to-end (now Phase 2) can't run until the plugin is installed. Phase 0 (skills-side plumbing) and Phase 1 (docs-side plugin) can run in parallel.
• Data flow is now offline-first — skills workflow checks out and builds the docs repo locally, reading from build/ directory. No network calls to fetch docs content. Local contributors point at a sibling docs checkout via --docs-path flag or DOCS_PATH env var. CI cost is a full Docusaurus build per sync; optimizations (caching) deferred.
• Provenance moved from HTML comments to YAML frontmatter — replaces <!-- generated-from: ... --> HTML comments with a metadata block in the rule's frontmatter (mode, sources, sourceCommit, inputHash). Reasoning: frontmatter is structured, already parsed by gray-matter, and matches the existing SKILL.md metadata pattern.
• Lock file removed — rules.manifest.lock.json is gone. Per-rule input hash now lives in the rule's own frontmatter, removing a sync-concern between two files.
• Validation Layer restructured — was a flat list of checks; now four named layers (skill schema → manifest lint → manifest↔frontmatter reconciliation → per-mode body checks) with an explicit per-mode applicability matrix. Layer 3 is the gate that makes the manifest causally authoritative.
• Open Questions — each question now annotated with the earliest phase it blocks (Phase 1, Phase 2, or Phase 4), replacing the previous flat "before Phase 1 begins" framing.

What was removed

• Alternative: Pointer Strategy section — entirely removed. direct mode subsumes its purpose (offline-safe, deterministic, no LLM call, no variance) without the operational complexity of submodules / sparse checkouts.

What's unchanged

• Goal, Guiding Principle ("Humans own the rule taxonomy; automation owns keeping rule bodies in sync").
• Rule / Skill / AGENTS.md / Manifest concepts and the rule-vs-skill distinction.
• Stories 1–5 (still illustrative; lightly updated to reflect new data-flow and metadata locations).
• Developer Documentation deliverable (expanding .github/CONTRIBUTING.MD).
• Migration philosophy (Phase 0 lands as no-behavior-change; progressive flip from synthesized to source-backed modes per rule).

[github-actions]

🎉 This PR is included in version 1.4.8 🎉

The release is available on GitHub release

Your semantic-release bot 📦🚀