From a2bb1d750ca4aad97c8c64b42575c218e0708d73 Mon Sep 17 00:00:00 2001 From: Neo Date: Mon, 13 Jul 2026 18:09:46 +0000 Subject: [PATCH] feat(examples): add skill-rl for training models on markdown instructions Introduces a new example showing how to train small language models to follow structured markdown skill definitions using reinforcement learning. Key components: - train_skill.py: Core training loop with RULER-based rewards - skills/git-commit: Conventional commit message generator skill - skills/code-review: Structured code review skill The reward function evaluates: - Format compliance (30%): Does output match specified format? - Procedure adherence (40%): Were all steps followed in order? - Correctness (30%): Is the output actually useful? This enables training 3B models to match or exceed 70B+ model performance on specific, well-defined tasks. --- examples/skill-rl/README.md | 239 +++++++++++++++++ examples/skill-rl/skills/code-review/SKILL.md | 138 ++++++++++ .../skill-rl/skills/code-review/tasks.jsonl | 15 ++ examples/skill-rl/skills/git-commit/SKILL.md | 112 ++++++++ .../skill-rl/skills/git-commit/tasks.jsonl | 30 +++ examples/skill-rl/train_skill.py | 248 ++++++++++++++++++ 6 files changed, 782 insertions(+) create mode 100644 examples/skill-rl/README.md create mode 100644 examples/skill-rl/skills/code-review/SKILL.md create mode 100644 examples/skill-rl/skills/code-review/tasks.jsonl create mode 100644 examples/skill-rl/skills/git-commit/SKILL.md create mode 100644 examples/skill-rl/skills/git-commit/tasks.jsonl create mode 100644 examples/skill-rl/train_skill.py diff --git a/examples/skill-rl/README.md b/examples/skill-rl/README.md new file mode 100644 index 000000000..e3c220a46 --- /dev/null +++ b/examples/skill-rl/README.md @@ -0,0 +1,239 @@ +# Skill-RL: Train Models to Follow Markdown Instructions + +Train small language models to perfectly execute structured markdown skills using reinforcement learning. + +## The Problem + +Large models (70B+) can follow complex markdown instructions reasonably well. Smaller models (3B-7B) struggle with: +- Multi-step procedures +- Conditional logic ("if X, then do Y") +- Output format compliance +- Tool calling sequences + +**Solution:** Use ART to train small models on specific skills until they match or exceed large model performance. + +## What is a "Skill"? + +A skill is a structured markdown document that defines: +1. **When to apply** — trigger conditions +2. **How to execute** — step-by-step procedure +3. **Expected output** — format and content requirements +4. **Verification** — how to check success + +Example skill structure: +```markdown +# SKILL.md — Git Commit Message Generator + +## Trigger +User asks to write a commit message for staged changes. + +## Procedure +1. Run `git diff --staged` to see changes +2. Identify the TYPE: feat|fix|docs|refactor|test|chore +3. Summarize the change in <50 chars for the subject +4. Add body paragraphs if change is complex +5. Reference issue numbers if mentioned + +## Output Format +``` +(): + + + +