Add deepset/roberta-large-squad2 question-answering recipe

[ssss141414]

Summary

Adds verified CPU/fp32 and CPU/fp16 recipes for deepset/roberta-large-squad2 extractive question answering. The contribution is Effort L0 / Outcome L0 and reaches Goal L3 on both required CPU tuples, with measured perf, PyTorch parity, SQuAD v2 task metrics, and component/op-level analysis. Current main already reaches L3 for CPU/fp32; these recipes add auditable tuple coverage plus checkpoint-specific evaluation metadata rather than masking an auto-config failure.

Model metadata

Frozen model-breakdown report: temp/pr1100_skill_run/model_breakdown_roberta_large_squad2.json; SHA-256 607d12fdea398260f67d551cd9007f5466058b387297fc46fec3d5ebd27d05a6. Markdown report: temp/pr1100_skill_run/model_breakdown_roberta_large_squad2.md.

What the model does

An English RoBERTa-large extractive-question-answering checkpoint that maps a question and context to start and end token-position logits, including the SQuAD 2.0 unanswerable-question setting.
Confidence: verified. Evidence: Hugging Face checkpoint config and model card at pinned revision 78fb38a59ea3cb6902e04d96da93efc87aeeff76; temp/pr1100_skill_run/04_inspect.json; Transformers v4.57.6 RobertaForQuestionAnswering.forward.

Primary user stories

• A user supplies a natural-language question and a passage to obtain the most likely answer span from that passage. Confidence: verified. Evidence: Checkpoint model-card Transformers pipeline example.
• A document-QA application supplies a question and candidate documents to extract answers while retaining the ability to reject unanswerable questions. Confidence: verified. Evidence: Checkpoint model-card Haystack example and SQuAD 2.0 description.

Supported tasks

Task	Support surfaces	Evidence	Confidence
question-answering	checkpoint, transformers, optimum-onnx, winml	Checkpoint model-index task=question-answering; config architectures=[RobertaForQuestionAnswering]; temp/pr1100_skill_run/03_metadata_probe.json; temp/pr1100_skill_run/04_inspect.json	verified

Model architecture

RobertaForQuestionAnswering uses a pooler-free RoBERTa encoder: embedding block, 24 repeated encoder layers with 16-head self-attention and 1024-to-4096-to-1024 GELU feed-forward blocks, followed by a per-token 1024-to-2 linear span head split into start_logits and end_logits.
Confidence: verified. Components: embeddings, encoder.layers[], encoder.layers[].attention, encoder.layers[].ffn, qa_outputs. Evidence: Pinned checkpoint config: 24 layers, hidden_size=1024, 16 heads, intermediate_size=4096; Transformers v4.57.6 modeling_roberta.py class construction and forward paths.

Validation and support evidence

Baseline

• Current-main commit: 76d48ad38f894c3fc5a5e760de67f0a096e438a6
• WinML: winml, version 0.2.0
• Optimum probe (after importing Optimum ONNX model configs): VENDOR-ONLY; vendor ['feature-extraction', 'fill-mask', 'multiple-choice', 'question-answering', 'text-classification', 'token-classification']; after WinML registration ['feature-extraction', 'fill-mask', 'multiple-choice', 'question-answering', 'text-classification', 'token-classification']; added by WinML [].
• Build: PASS — Build complete in 137.0s; final artifact temp/pr1100_baseline_build/model.onnx; 1325 ONNX nodes; 100% hierarchy-tag coverage.
• Perf: CPUExecutionProvider fp32: mean 881.561 ms, p50 889.987 ms, 1.13 samples/s, RSS model-load delta 1166.62 MB.
• Eval: rajpurkar/squad_v2 validation, 100 shuffled samples, seed 42: exact 85.0, F1 88.70079365079366, 1.1534829273985954 samples/s.
• Baseline Goal floor: L3
• Starting auto-config recipe: C:/repo/winml-cli-pr1100-baseline/temp/pr1100_baseline (generated config file)

Baseline commands:

uv run winml build -m deepset/roberta-large-squad2 -o temp/pr1100_baseline_build --ep cpu --device cpu --no-analyze --no-optimize --no-quant --no-compile --rebuild
uv run winml perf -m temp/pr1100_baseline_build/model.onnx --task question-answering --ep cpu --device cpu --precision fp32 --iterations 20 --warmup 5 --no-monitor --memory -o temp/pr1100_skill_run/26_refresh_baseline_perf_fp32.json --overwrite -f json --no-color
uv run winml eval -m temp/pr1100_baseline_build/model.onnx --model-id deepset/roberta-large-squad2 --task question-answering --dataset rajpurkar/squad_v2 --split validation --samples 100 --column question_column=question --column context_column=context --column id_column=id --column label_column=answers --ep cpu --device cpu -o temp/pr1100_skill_run/27_refresh_baseline_eval_fp32.json --overwrite -f json --no-color
uv run winml config -m deepset/roberta-large-squad2 -t question-answering -o temp/pr1100_baseline

Goal

• Effort: L0
• Committed Goal ceiling: L3
• Outcome: L0
• Success definition: L0 build/structure; L1 measured perf; L2 PyTorch parity; L3 SQuAD v2 task metrics, for CPU fp32 and fp16.
• Precision plan: CPU/cpu fp32 and fp16; no tuple omitted.

Outcome

• Highest Goal verdict: L3 PASS
• Coverage: full; deferred tuples: []
• Shipped recipes:
  • examples/recipes/deepset_roberta-large-squad2/cpu/cpu/question-answering_fp32_config.json
  • examples/recipes/deepset_roberta-large-squad2/cpu/cpu/question-answering_fp16_config.json
• Code paths: none (recipe-only L0).
• Model finding: roberta-001 in copilot-skills/dev_skill/adding-model-support/model_knowledge/roberta.json (Lane A skills branch).
• Methodology friction observed: _meta-077 and reviewer-loop _meta-078 are committed and pushed through 78231091 on the required separate ModelKitArtifacts Lane A branch. They record schema/fp16 handoff validation and the lint/methodology gate contradiction found by independent review. Skill files cannot be mixed into this winml-cli L0 model PR.

Per-EP/device/precision results — including perf and eval data

Goal ladder

Tier	CPU/fp32	CPU/fp16
L0 build + structure	PASS — 136.9 s; FLOAT initializers; 1,419,341,824-byte external data	PASS — 153.2 s; 396 FLOAT16 initializers; 709,670,912-byte external data
L1 perf	PASS — mean 902.339 ms; p50 904.426 ms	PASS — mean 1041.909 ms; p50 1040.757 ms
L2 HF parity	PASS — mean cosine start/end 0.9999999999794873 / 0.9999999999866034; mean max-abs 0.00038888752460479735 / 0.0003369855880737305	PASS — mean cosine start/end 0.9999928808311433 / 0.9999945743800909; mean max-abs 0.22941139101982116 / 0.20585397601127625
L3 SQuAD v2	PASS — exact 85.0; F1 88.70079365079366	PASS — exact 85.0; F1 88.70079365079366

Perf

EP / device	Precision	Verdict	Mean	p50	Throughput	RAM model-load Δ	RAM total Δ	VRAM local total Δ
CPUExecutionProvider / cpu	fp32	PASS	902.339 ms	904.426 ms	1.11 samples/s	1167.0 MB	1302.27 MB	0.0 MB
CPUExecutionProvider / cpu	fp16	PASS	1041.909 ms	1040.757 ms	0.96 samples/s	1180.8 MB	1347.24 MB	0.0 MB

Eval

EP / device	Precision	Verdict	Dataset	Revision	Subset	Metrics	Throughput
CPUExecutionProvider / cpu	fp32	PASS	rajpurkar/squad_v2	None	validation; 100 shuffled samples; seed=42	exact 85.0; F1 88.70079365079366	1.0662427603903848 samples/s
CPUExecutionProvider / cpu	fp16	PASS	rajpurkar/squad_v2	None	validation; 100 shuffled samples; seed=42	exact 85.0; F1 88.70079365079366	0.9097509714650179 samples/s

Delta

• Recipe comparison: CHANGED — Canonical JSON comparison after omitting main auto-config /compile:null: both shipped recipes add only /eval; fp32 and fp16 recipe payloads are identical because precision is selected by the CLI flag and filename coverage claim.
• Relative to current-main auto-config, both shipped recipes add only /eval: task question-answering, dataset rajpurkar/squad_v2, validation split, 100 samples, and the question/context/id/answers column mapping. Auto-config's /compile: null omission is not a behavior change.
• The /eval delta is checkpoint-specific human intent and is not safely derivable as a class-wide RoBERTa rule.
• Recipe-free acceptance: PASS — Current-main build completed in 137.0s. The recipe is a verified CPU coverage and checkpoint-specific eval declaration, not a workaround for an auto-config build failure.
• Reducibility consistent with charter: True.
• examples/recipes/README.md remains untouched.

Analyze results — component level and op level

Command status: PASS, exit code 0. Artifact: temp/pr1100_skill_run/17_analyze_all_fp32.json. Error: None.

Component-level analysis

Mapped report: temp/pr1100_skill_run/model_breakdown_roberta_large_squad2_mapped.json; SHA-256 288c9a6529df2cba5a603af2aa95f3bfc9b67471a92f98998e9dde023f099d4e.

Component	ONNX region	Node count	Operator counts	Basis / confidence	Per-EP partial or unsupported op types
embeddings	13 hierarchy-tagged nodes under the component's PyTorch-derived ONNX scopes	13	Gather 2, Add 3, Cast 3, Equal 1, Not 1, CumSum 1, Mul 1, LayerNormalization 1	scope-name / mapped	QNNExecutionProvider: partial OP/ai.onnx/Add, OP/ai.onnx/Mul, unsupported none
encoder.layers[]	840 hierarchy-tagged nodes under the component's PyTorch-derived ONNX scopes	840	MatMul 192, Add 240, Reshape 96, Transpose 96, Mul 96, Softmax 24, LayerNormalization 48, Div 24, Erf 24	scope-name / mapped	QNNExecutionProvider: partial OP/ai.onnx/Div, OP/ai.onnx/Erf, OP/ai.onnx/Add, OP/ai.onnx/Mul, unsupported none
encoder.layers[].attention	576 hierarchy-tagged nodes under the component's PyTorch-derived ONNX scopes	576	MatMul 144, Add 144, Reshape 96, Transpose 96, Mul 48, Softmax 24, LayerNormalization 24	scope-name / mapped	QNNExecutionProvider: partial OP/ai.onnx/Add, OP/ai.onnx/Mul, unsupported none
encoder.layers[].ffn	264 hierarchy-tagged nodes under the component's PyTorch-derived ONNX scopes	264	MatMul 48, Add 96, Div 24, Erf 24, Mul 48, LayerNormalization 24	scope-name / mapped	QNNExecutionProvider: partial OP/ai.onnx/Div, OP/ai.onnx/Erf, OP/ai.onnx/Add, OP/ai.onnx/Mul, unsupported none
qa_outputs	2 hierarchy-tagged nodes under the component's PyTorch-derived ONNX scopes	2	MatMul 1, Add 1	scope-name / mapped	QNNExecutionProvider: partial OP/ai.onnx/Add, unsupported none

Unmapped: 10 of 865 nodes — graph-input preparation and final split/squeeze plumbing remain explicit gaps rather than heuristic component assignments.

Op-level analysis

• Total operators: 865; unique operator types: 20.
• Counts: Gather 2, Add 244, Cast 5, Equal 1, Not 1, CumSum 1, Mul 97, LayerNormalization 49, MatMul 193, Reshape 96, Transpose 96, Unsqueeze 2, Expand 1, Sub 1, Where 1, Softmax 24, Div 24, Erf 24, Split 1, Squeeze 2.
• Classification normalization: when analyzer rule output lists an op as both supported and partial, partial takes precedence in this table; raw arrays remain preserved in verdict_table.json. CPU has no rule-backed runtime support, so its 20 op types are explicitly unknown rather than omitted.

IHV / EP / device	Supported	Partial	Unsupported	Unknown
NVIDIA / NvTensorRTRTXExecutionProvider / GPU	20	none	none	0
NVIDIA / CUDAExecutionProvider / GPU	0	none	none	20
AMD / MIGraphXExecutionProvider / GPU	0	none	none	20
QC / QNNExecutionProvider / GPU	16	OP/ai.onnx/Div, OP/ai.onnx/Erf, OP/ai.onnx/Add, OP/ai.onnx/Mul	none	0
Intel / OpenVINOExecutionProvider / GPU	20	none	none	0
Microsoft / DmlExecutionProvider / GPU	0	none	none	20
Microsoft / CPUExecutionProvider / CPU	0	none	none	20

Reproduce commands

uv run winml build -c examples/recipes/deepset_roberta-large-squad2/cpu/cpu/question-answering_fp32_config.json -m deepset/roberta-large-squad2 -o temp/pr1100_skill_run/tested/fp32 --ep cpu --device cpu --precision fp32 --no-analyze --no-quant --no-compile --rebuild --no-color
uv run winml build -c examples/recipes/deepset_roberta-large-squad2/cpu/cpu/question-answering_fp16_config.json -m deepset/roberta-large-squad2 -o temp/pr1100_skill_run/tested/fp16 --ep cpu --device cpu --precision fp16 --no-analyze --no-compile --rebuild --no-color
uv run winml perf -m temp/pr1100_skill_run/tested/fp32/model.onnx --task question-answering --ep cpu --device cpu --precision fp32 --iterations 20 --warmup 5 --memory --no-monitor -o temp/pr1100_skill_run/13_perf_fp32.json --overwrite -f json --no-color
uv run winml perf -m temp/pr1100_skill_run/tested/fp16/model.onnx --task question-answering --ep cpu --device cpu --precision fp16 --iterations 20 --warmup 5 --memory --no-monitor -o temp/pr1100_skill_run/14_perf_fp16.json --overwrite -f json --no-color
uv run winml eval -m temp/pr1100_skill_run/tested/fp32/model.onnx --model-id deepset/roberta-large-squad2 --task question-answering --mode compare --ep cpu --device cpu -o temp/pr1100_skill_run/18_compare_fp32.json --overwrite -f json --no-color
uv run winml eval -m temp/pr1100_skill_run/tested/fp16/model.onnx --model-id deepset/roberta-large-squad2 --task question-answering --mode compare --ep cpu --device cpu -o temp/pr1100_skill_run/19_compare_fp16.json --overwrite -f json --no-color
uv run winml eval -m temp/pr1100_skill_run/tested/fp32/model.onnx --model-id deepset/roberta-large-squad2 --task question-answering --dataset rajpurkar/squad_v2 --split validation --samples 100 --column question_column=question --column context_column=context --column id_column=id --column label_column=answers --ep cpu --device cpu -o temp/pr1100_skill_run/15_eval_fp32.json --overwrite -f json --no-color
uv run winml eval -m temp/pr1100_skill_run/tested/fp16/model.onnx --model-id deepset/roberta-large-squad2 --task question-answering --dataset rajpurkar/squad_v2 --split validation --samples 100 --column question_column=question --column context_column=context --column id_column=id --column label_column=answers --ep cpu --device cpu -o temp/pr1100_skill_run/16_eval_fp16.json --overwrite -f json --no-color
uv run winml analyze -m temp/pr1100_skill_run/tested/fp32/model.onnx --ep all --output temp/pr1100_skill_run/17_analyze_all_fp32.json --format json --no-color

[ssss141414]

Independent skill-workflow verdict: APPROVE

A separate read-only reviewer pass verified current main 76d48ad38f894c3fc5a5e760de67f0a096e438a6 against PR head 40e9971b2398ce32567fe0f9ecbf373b5f98359e.

• Diff scope is exactly the two CPU recipe JSON files.
• CPU/fp32 and true CPU/fp16 pass L0-L3 with measured perf, HF parity, and 100-sample SQuAD v2 eval data.
• True fp16 is evidenced by 396 FLOAT16 initializers and 709,670,912-byte external data; the fp16 build does not combine --precision fp16 with --no-quant.
• The refreshed baseline is supported: build 137.0 s; fp32 mean/p50 881.561/889.987 ms; exact 85.0; F1 88.70079365079366.
• Component-level and op-level analysis are both present.
• No new ruff regression is introduced; unrelated failures already on main are outside this JSON-only diff.
• Lane A findings roberta-001, _meta-077, and _meta-078 are present at pushed ModelKitArtifacts commit 78231091 without crossing repository boundaries.
• The model-breakdown subskill handoff and planner/producer/tester/learner/explainer/reviewer responsibility chain are complete after the recorded contract corrections.

No blocking issues. A formal GitHub APPROVE attempt from the independent Enterprise Managed User account was rejected by GitHub authorization policy, so this structured verdict is posted as a comment. The PR intentionally remains draft.