Annotate Dolci data with propella

[geoalgo]

We want to annotate dolci data with propella annotations for dataset selection and other purposes.

First we would want to analyse qualitatively whether the annotation makes sense for instruction data (propella was validated for pretraining raw data).

Then, we want to estimate the compute required before completing the annotation.

[KonstiNik]

Qualitative inspection of Propella annotations on Dolci data

I ran Propella-1-4b locally to check whether the annotations make sense for instruction data.
As a first step, I tested on 7 handcrafted example documents covering different content types (code, news, ads, academic text, etc.). As a second step, I ran it on a sample of the Dolci-Instruct-SFT dataset (1 sample per domain, 8 total) to test on real instruction data.

Setup: llama.cpp on Apple Silicon (Metal), BF16, with strict JSON schema enforcement via GBNF grammar. The pipeline and scripts are in propella-local.

Results (comparison tables):
Handcrafted examples vs. expected — 7 handcrafted documents of varying types
Dolci annotations vs. expected — 8 documents across all Dolci domains (Chat, Coding, Math, Multilingual, Other, Precise IF, Safety, Science)
Green = model matches expected, red = mismatch. Expected annotations were generated independently with Claude Opus 4.6.

Initial observations:

• The model produces valid, schema-complete annotations for all tested instructions without errors
• Where the model and Claude Opus disagree, both answers are generally plausible — many annotation properties are inherently ambiguous, and differences reflect reasonable interpretations rather than clear errors
• Multilingual content (Yoruba, German, Arabic) is handled without issues
• Overall the annotations look usable for instruction data

Next step would be estimating compute for the full dataset.

[KonstiNik]

Compute estimation by existing numbers:

Model: propella-1-4b

Datasets:
Dolci Instruct SFT: 2.15M samples
Dolci Think SFT 32B: 2.25M samples

GPU	Docs/s	Docs/h	GPUh Instruct	GPUh Think
A100	10.3	37k	2.15M / 37k = 58	2.25M / 37k = 61
H100	27	97k	2.15M / 97k = 22	2.25M / 97k = 23

Dividing by 4 GPUs, each dataset takes ~15H (A100) or 6h (H100). So it's actually quite fast.

Next: Numbers should be verified by running a small benchmark.