Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions docs/evals-and-specs/evaluations.md
Original file line number Diff line number Diff line change
Expand Up @@ -187,7 +187,7 @@ Most commonly, you'll want to populate the datasets using synthetic data. Clicki

We suggest at least 160 data samples per eval. Difficult or subjective tasks may require more.

An appropriate data gen template will be populated when you enter data-gen via an eval. You can customize this template to guide data generation. See [the docs](../synthetic-data-generation.md#automatic-templates-and-custom-prompting) for details.
An appropriate data gen template will be populated when you enter data-gen via an eval. You can customize this template to guide data generation. See [the docs](../synthetic-data-generation.md#templates-and-custom-guidance) for details.

{% hint style="info" %}
Golden eval datasets work best if they have a range of ratings (some pass, some fail, some of each star-score).
Expand Down Expand Up @@ -337,7 +337,7 @@ Like mean squared error, but scores are normalized to the range 0-1. For example
If you see "N/A" scores in your correlation table, it means more data is needed. This can be one of two cases

* _**Simply not enough data**_: if your golden dataset is very small (<10 items) it can be impossible to produce confident correlation scores. Add more data to resolve this case.
* _**Not enough variation of human ratings in the golden dataset**_: if you have a larger dataset, but still get N/A, it's likely there isn't enough variation in your dataset for the given score. For example, if all of the golden samples of a score pass, the evaluator won't produce a confident correlation score, as it has no failing examples and everything is a tie. Add more content to your golden dataset, designing the content to fill out the missing score ranges. You can use synthetic data gen [human guidance](../synthetic-data-generation.md#human-guidance) to generate examples that fail.
* _**Not enough variation of human ratings in the golden dataset**_: if you have a larger dataset, but still get N/A, it's likely there isn't enough variation in your dataset for the given score. For example, if all of the golden samples of a score pass, the evaluator won't produce a confident correlation score, as it has no failing examples and everything is a tie. Add more content to your golden dataset, designing the content to fill out the missing score ranges. You can use synthetic data gen [human guidance](../synthetic-data-generation.md#templates-and-custom-guidance) to generate examples that fail.

</details>

Expand Down
2 changes: 1 addition & 1 deletion docs/fine-tuning/fine-tuning-guide.md
Original file line number Diff line number Diff line change
Expand Up @@ -182,7 +182,7 @@ Models and products are rarely perfect on their first try. When you find bugs or
* Experiment with fine-tuning hyperparameters (see the "Advanced Options" section of the UI)
* Experiment with shorter training prompts, which can reduce costs
* For one-off bugs you encounter use Kiln to “[repair](../repairing-responses.md)” the issues. These get added to your training data for future iterations.
* For recurring bugs/patterns, use [synthetic data generation](../synthetic-data-generation.md) to generate many samples of common bugs, ensure they have correct responses with [human guidance](../synthetic-data-generation.md#human-guidance), and add the results to the training set to prevent this class of issues in the future.
* For recurring bugs/patterns, use [synthetic data generation](../synthetic-data-generation.md) to generate many samples of common bugs, ensure they have correct responses with [human guidance](../synthetic-data-generation.md#templates-and-custom-guidance), and add the results to the training set to prevent this class of issues in the future.
* Rate your dataset using Kiln’s [rating system](../reviewing-and-rating.md), then build fine-tunes using only highly rated content.
* Regenerate fine-tunes as your dataset grows and evolves
* Try new foundation models (directly and with fine tuning) when new state of the art models are released.
Expand Down
Loading