Skip to content

Rewrite synthetic data gen docs for Kiln Pro batch planning#13

Open
sfierro wants to merge 2 commits into
mainfrom
sfierro/batch-gen-docs
Open

Rewrite synthetic data gen docs for Kiln Pro batch planning#13
sfierro wants to merge 2 commits into
mainfrom
sfierro/batch-gen-docs

Conversation

@sfierro

@sfierro sfierro commented Jul 18, 2026

Copy link
Copy Markdown
Contributor

Updates the Synthetic Data Generation docs for the new Kiln Pro batch planning flow, and corrects drift in the existing (manual) content. Behavior was audited against the Kiln app source rather than inferred.

Accuracy fixes

Verified against the Kiln repo:

  • Fine-tuning split tag is fine_tune_data, not fine_tuning_data
  • Eval split tag names come from the eval's own tag filters; only the 80/20 ratio is fixed
  • Eval templates were wrong. The docs claimed two options (Issue, Requirements) that you pick from. In reality there are four eval-derived templates (Requirements Eval, Desired Behaviour Eval, Appropriate Tool Use, Issue Eval — legacy), and Kiln auto-selects the one matching your eval's type
  • Documented the six built-in templates (Fine Tuning, Toxicity, Bias, Maliciousness, Jailbreak, Factual Correctness)
  • Refreshed model recommendations, which named models that are now out of date
  • Noted the Data Guide applies at the topic and input stages only, not to output generation

Structure

The previous revision inserted the Pro flow alongside the untouched manual content, which left the two flows reading as unrelated and duplicated their shared steps.

  • Both flows are now framed as the same four stages, with only the first differing (Kiln Pro plans the batch; manual builds a topic tree). Inputs, outputs, and saving are documented once instead of twice
  • Templates and custom guidance are scoped to the manual flow. The template dropdown and per-stage guidance are manual-only in the app; Kiln Pro uses a single prefilled plan-level Guidance box. The old shared section implied they worked the same way
  • Data gen model choice moved ahead of the build flows
  • Clarified that Manual refers to how coverage is decided, not to hand-authoring data (matching the app's "Manual vs AI Planned" coverage framing)

Link fixes

  • evaluations.md and fine-tuning-guide.md repointed to the current section
  • Two of those links pointed at a #human-guidance anchor that did not exist and were already broken before this change
  • All internal and external anchors into this page verified to resolve

Notes for review

  • A few <!-- TODO IMAGE --> / <!-- TODO VIDEO --> placeholders mark screenshots still needed for the Pro flow
  • The "Resolving Bugs & Issues" recipe was softened from "create an eval using the Issue template" to a template-neutral phrasing, since the Issue template is now legacy-only. Worth a look if there's a current canonical path there

🤖 Generated with Claude Code

sfierro and others added 2 commits July 17, 2026 21:03
Rewrite the Set Up A Data Guide section for the v2 flow: Manual vs
Kiln Pro chooser, the Kiln Pro copilot path (connect, add example
inputs, generation settings, draft), and the shared Review and Refine
cycle. Remove references to output examples, the output model picker,
and applying the guide at output generation, which no longer exist.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Audit the Kiln app's synthetic data generation behavior and rewrite the
docs to match, restructuring around the two build flows.

Accuracy fixes (verified against the Kiln repo):
- Fine-tuning split tag is fine_tune_data, not fine_tuning_data
- Eval split tag names come from the eval's own tag filters; only the
  80/20 ratio is fixed
- Correct the eval template section: there are four eval-derived
  templates (Requirements Eval, Desired Behaviour Eval, Appropriate Tool
  Use, Issue Eval), auto-selected from the eval's type rather than
  picked from a menu of two
- Document the six built-in templates
- Refresh out-of-date model recommendations
- Note the Data Guide applies at the topic and input stages only

Structure:
- Frame both flows as the same four stages, with only the first stage
  differing (Kiln Pro plans the batch; manual builds a topic tree), so
  inputs, outputs, and saving are documented once instead of twice
- Scope templates and custom guidance to the manual flow, since the
  template dropdown and per-stage guidance are manual-only; Kiln Pro
  uses a single prefilled plan-level Guidance box
- Move data gen model choice ahead of the build flows
- Clarify that Manual refers to how coverage is decided, not to
  hand-authoring data

Also repoint links in evaluations.md and fine-tuning-guide.md, including
two that pointed at a #human-guidance anchor that did not exist.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant