Add audeering wav2vec2 dimensional emotion (speech regression) support#1084
Open
DingmaomaoBJTU wants to merge 6 commits into
Open
Add audeering wav2vec2 dimensional emotion (speech regression) support#1084DingmaomaoBJTU wants to merge 6 commits into
DingmaomaoBJTU wants to merge 6 commits into
Conversation
Collaborator
Author
REVIEWER verdict: REQUEST_CHANGESI independently re-ran the reviewer checklist from PR head Blocking items (producer action required)
Checked evidence
Reachable verified EP: |
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
DingmaomaoBJTU
force-pushed
the
add-audeering-wav2vec2-emotion
branch
from
da4d514 to
f3b531a
Compare
Collaborator
Author
|
Addressed all 4 items in f3b531a: rebased onto origin/main 130acfe, renamed recipe fp16→fp32 (0 FLOAT16 initializers confirmed), added baseline-FAIL gate evidence to item 7, re-ran CPU perf (representative median avg 381.46 ms across runs 377.25, 380.48, 382.43, 401.01 ms; variance noted). Ready for re-review. |
Collaborator
Author
|
APPROVE Verified PR #1084 from fresh PR head Checkout / scope
Engineering/code review
PR body / report audit
Independent rebuild and artifact verification
Goal ladder re-verification
Baseline gate
Prior REQUEST_CHANGES items
EP coverage annotation
Sign-off
|
nn.Linear.__call__ is typed to return Any, so
eturn x tripped the strict
mypy [no-any-return] gate (CI lint). Return the projection via
cast("torch.Tensor", ...) to match the repo house style (blip.py/mu2.py) and
drop the now-unnecessary RET504 assign-then-return. Runtime behavior unchanged.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
xieofxie
reviewed
Jul 10, 2026
xieofxie
left a comment
xieofxie
left a comment
Contributor
There was a problem hiding this comment.
Nice, clean addition — the routing follows house style well: hyphenated MODEL_CLASS_MAPPING keys that rely on the resolver's _→- normalization (consistent with the qwen3-* variants), register_specialization(..., "WinMLModelForGenericTask"), and the model_type_override plumbing via the recipe loader block. Lint passes and the RegressionHead/mean-pool forward match the audeering reference, which lines up with the L2 cosine ≈1.0 you reported.
My main ask is test coverage — see the inline note on wav2vec2.py. Every comparable model-class/ONNX-config addition ships a small unit test, and this one doesn't. The rest are minor nits.
- Add tests/unit/models/wav2vec2/test_onnx_config.py covering the MODEL_CLASS_MAPPING entry -> EmotionModel, the resolve_task underscore (_ -> -) normalization contract, and the IOConfig registration plus input/output axes. - Add the '# triggers registration' comment to the IOConfig side-effect import and collapse the mapping import to a single statement. - Restore the fp16/all-buckets guarantee wording in the recipes README and note the new fp32/CPU-only exception instead of weakening every row. - Drop the unused **kwargs from RegressionHead.forward.
…validation test Relocate the fp32 CPU-tested recipe to examples/recipes/audeering_wav2vec2-large-robust-12-ft-emotion-msp-dim/cpu/cpu/, matching the <model>/<ep>/<device> layout convention. Add tests/unit/recipes/test_cpu_recipes.py to validate the recipe loads and routes to the emotion-regression head.
Keep the README unchanged from main: audeering/wav2vec2-large-robust-12-ft-emotion-msp-dim is fp32/CPU-only and does not satisfy the table's fp16-on-all-buckets invariant, so it is not listed there.
Per CPU precision policy: the cpu/cpu bucket ships both float precisions (fp32 + fp16, quant:null). Adds audio-classification_fp16_config.json (byte-identical to the existing fp32 recipe) and a matching parametrized entry in tests/unit/recipes/test_cpu_recipes.py. Verified: test_cpu_recipes.py 2 passed (fp32 + fp16); fp16 recipe SHA256 identical to the already-verified fp32 recipe. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
DingmaomaoBJTU
added a commit
that referenced
this pull request
Jul 16, 2026
…cation, NLI) (#1117) ## Summary Adds a curated CPU float recipe for **`cross-encoder/nli-deberta-v3-base`** — a DeBERTa-v3 cross-encoder for natural-language inference (3-way sequence classification: contradiction / entailment / neutral; config `model_type = deberta-v2`). Ships **fp32 + fp16** variants under `cpu/cpu/` (both `quant: null`; no CPU quantized variant per repo convention). This is a **recipe-only (L0★)** contribution: Optimum already covers `deberta-v2` `text-classification` natively, so `main` builds this model with zero source changes. The delta this PR adds is the **curated CPU reference recipe** plus an **L2 numeric-parity proof** that the trained 3-way NLI head is preserved end-to-end (which a plain build does not demonstrate). Claimed tiers: **Effort L0★ · Goal ceiling L2 · Outcome L0**. --- ### 1. Recipe path(s) - `examples/recipes/cross-encoder_nli-deberta-v3-base/cpu/cpu/text-classification_fp32_config.json` - `examples/recipes/cross-encoder_nli-deberta-v3-base/cpu/cpu/text-classification_fp16_config.json` Both are byte-identical (`quant: null` float bucket; on CPU both realize as fp32, fp16 materializes on GPU/NPU). opset 17, batch 1, inputs `input_ids[1,512]` + `attention_mask[1,512]` (int32), output `logits`. No `token_type_ids` — `deberta-v2` has `type_vocab_size = 0`. ### 2. README row None. Recipe-only, CPU-only — deliberately **not** added to the "Total: N (model, task) tuples that pass fp16 eval on all 10 (EP, device) buckets" table, which would be a factual overclaim for a CPU-only recipe (consistent with the reviewed outcome on #1084 / #1112). ### 3. Build output dir `temp/nli_deberta_v3/` (scratch, gitignored) — `model.onnx` + `model.onnx.data` (786 MB fp32). ### 4. Build log `✅ Build complete` — recipe-driven `winml build` exit 0 (269.5 s). ONNX IR 8, opset 17, external data co-located, output `logits[1,3]`. ### 5. Appended findings `model_knowledge/deberta.json` → `deberta-001` (new family). Lane A (skill repo); **not** part of this model PR's diff. ### 6. Optimum-coverage probe `deberta-v2` `text-classification` is **VENDOR-ONLY** (`added_by_winml = []`). Vendor onnx tasks: `[feature-extraction, fill-mask, multiple-choice, question-answering, text-classification, token-classification]`. No winml-added exporter exists or is needed. ### 7. Claimed (Effort, Goal, Outcome) **L0★ / L2 / L0.** Baseline: `winml build -m cross-encoder/nli-deberta-v3-base` on `main` (`origin/main = 7434269`, winml 0.2.0) already **PASSES** build-only. Contribution = curated CPU float recipe (fp32 + fp16) + L2 trained-head-preservation proof. ### 8. Goal-ladder verdict table | Tier | Verdict | Evidence | |---|---|---| | L0 (build) | **PASS** | recipe-driven build exit 0 (269.5 s); ONNX opset 17, inputs `input_ids[1,512]`+`attention_mask[1,512]`, output `logits[1,3]`, external data co-located | | L1 (perf) | **PASS** | CPU/fp32 avg 5951 ms, throughput 0.17 samples/s, RAM Δ +618 MB (slow but deterministic; ContextPooler pools fixed position 0, so random-dummy perf is safe) | | L2 (numeric vs PyTorch) | **PASS** | 4 real NLI pairs: cosine **1.000000**, max-abs **1.9e-6**, argmax agreement **4/4** — trained 3-way NLI head + disentangled attention export losslessly | Ceiling L2 reached; no downgrade. ### 9. Methodology-evolution declaration No methodology friction this cycle. `deberta` is a new model family (new `model_knowledge/deberta.json`), but the existing pipeline handled it without a skill_meta change. ### 10. Perf & eval data | EP / Device | Precision | Verdict | Mean | p50 | Throughput | RAM Δ | Task metric | |---|---|---|---|---|---|---|---| | CPUExecutionProvider / cpu | fp32 | PASS | 5951 ms | — | 0.17 samples/s | +618 MB | N/A (L3 not marched — L2 ceiling) | `winml eval` not run (Goal ceiling L2). NLI label order for this checkpoint is `{0:contradiction, 1:entailment, 2:neutral}`, which differs from the GLUE/MNLI dataset order — an eval would need a label remap, so no eval block is shipped (mirrors `facebook/bart-large-mnli`). ### 11. Component / op-level data `winml analyze --ep all`: **568 total operators, 17 unique**. Per-EP op classification: **QNN NPU 17/17 supported**, **OpenVINO NPU 17/17 supported** (includes the disentangled-attention `GatherElements` ops); **VitisAI all-unknown** (no rule data — analyze exit 1, expected, not a functional failure). Artifact: `temp/nli_deberta_v3/analyze_all.json`. ### 12. Reproducible commands ```powershell # baseline (main already builds this model, no recipe) winml build -m cross-encoder/nli-deberta-v3-base -o temp\nli_deberta_v3_baseline # recipe-driven build (this PR) winml build -m cross-encoder/nli-deberta-v3-base ` -c examples\recipes\cross-encoder_nli-deberta-v3-base\cpu\cpu\text-classification_fp32_config.json ` -o temp\nli_deberta_v3 # L1 perf (CPU) winml perf -m temp\nli_deberta_v3\model.onnx --iterations 20 --warmup 5 --no-analyze # op coverage winml analyze -m temp\nli_deberta_v3\model.onnx --ep all -o temp\nli_deberta_v3\analyze_all.json # L2 parity vs PyTorch: temp\nli_deberta_v3_l2.py (4 real NLI pairs, pad to 512, int32 inputs, drop token_type_ids) python temp\nli_deberta_v3_l2.py ``` Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
xieofxie
reviewed
Jul 16, 2026
xieofxie
left a comment
xieofxie
left a comment
Contributor
There was a problem hiding this comment.
Re-reviewed at a9a1af5. All four points from my earlier pass are addressed — thanks:
- Unit tests now ship (
tests/unit/models/wav2vec2/test_onnx_config.py,tests/unit/recipes/test_cpu_recipes.py), locking in the mapping entry, the_→-model_type normalization, and the ONNX I/O axes. # triggers registrationcomment added to the side-effect import inmodels/hf/__init__.py.examples/recipes/README.mdreverted to keep the stronger guarantee.- Unused
**kwargsdropped fromRegressionHead.forward.
Verified locally at head: ruff clean on the changed files, and all 10 new tests pass. The wav2vec2.py module faithfully reproduces the audeering head/mean-pool, routing is architecture-driven (no hardcoded id logic), and the registration wiring is correct.
One remaining consistency question about the two recipe files — see inline. Non-blocking.
| ] | ||
| }, | ||
| "optim": {}, | ||
| "quant": null, |
Contributor
There was a problem hiding this comment.
This _fp32_config.json is byte-identical to its _fp16_config.json sibling (same git blob), and it's the only _fp32_config.json in the whole repo — the other 76 curated models each ship a single _fp16_config.json. Because both files set "quant": null, they produce the same output: with -c ..._fp16_config.json alone you still get an fp32 model, since fp16 conversion only kicks in when quant.mode == "fp16" (build.py:1377) or when --precision fp16 is passed on the CLI (which patches quant regardless of which of these two files you point at). So the fp16/fp32 filenames here imply a difference that doesn't exist, and your own test_cpu_recipes.py asserts quant is None for both.
Two consistent options:
- Drop the fp32 duplicate and keep just the
_fp16_config.json, matching the single-recipe-per-(model,task) convention (fp16 is what the README advertises; fp32 isn't in the variant taxonomy). - If a self-sufficient fp16 recipe is intended (so
-calone yields fp16 without--precision), give the fp16 file a realquantblock with"mode": "fp16"— but note no recipe in the repo currently does that, so option 1 is the lower-friction path.
Non-blocking, but shipping two identical files under different precision names is likely to confuse users.
ssss141414
added
the
model-scale-by-skill
DingmaomaoBJTU
added a commit
that referenced
this pull request
Jul 16, 2026
…ication, reranker) (#1118) ## Summary Adds a curated CPU float recipe for **`mixedbread-ai/mxbai-rerank-base-v1`** — a cross-encoder **reranker** that scores `(query, document)` relevance (config `model_type = deberta-v2`, `DebertaV2ForSequenceClassification`, `num_labels = 1` → a single relevance logit). Ships **fp32 + fp16** variants under `cpu/cpu/` (both `quant: null`; no CPU quantized variant per repo convention). This is a **recipe-only (L0★)** contribution: Optimum already covers `deberta-v2` `text-classification` natively, so `main` builds this model with zero source changes. The delta is the **curated CPU reference recipe** plus an **L2 numeric + ranking parity proof** that the trained reranker head is preserved end-to-end. Notably the recipe is **byte-identical** to the now-merged `cross-encoder/nli-deberta-v3-base` recipe (#1117, on `main` at `e95011d3`) — identical git blob OID `7b36e69520b418c7c86b2d8a39eab741cb9742c5` (git-stored SHA256 `8b587c4b…bec8`; verify with `git rev-parse HEAD:<path>`). The `model_id` is CLI-passed and the output width is config-driven, so one `deberta-v2` text-classification recipe serves the whole task family. Claimed tiers: **Effort L0★ · Goal ceiling L2 · Outcome L0**. > **Updated after review (#1118):** rebased onto current `origin/main` (`e95011d3`, which now includes the merged #1117 NLI recipe) and corrected the byte-identity evidence to git's canonical blob OID. (An earlier `Get-FileHash` value `59E337…` was the Windows **CRLF** working-tree hash; git normalizes to **LF** on commit, so the on-`main` content hash is `8b587c4b…`.) --- ### 1. Recipe path(s) - `examples/recipes/mixedbread-ai_mxbai-rerank-base-v1/cpu/cpu/text-classification_fp32_config.json` - `examples/recipes/mixedbread-ai_mxbai-rerank-base-v1/cpu/cpu/text-classification_fp16_config.json` Byte-identical (`quant: null` float bucket; on CPU both realize as fp32, fp16 materializes on GPU/NPU). opset 17, batch 1, inputs `input_ids[1,512]` + `attention_mask[1,512]` (int32), output `logits[1,1]`. No `token_type_ids` — `deberta-v2` has `type_vocab_size = 0`. ### 2. README row None. Recipe-only, CPU-only — deliberately **not** added to the "passes fp16 eval on all 10 (EP, device) buckets" table, which would be a factual overclaim for a CPU-only recipe (consistent with #1084 / #1112 / #1117). ### 3. Build output dir `temp/mxbai_rerank/` (scratch, gitignored) — `model.onnx` + `model.onnx.data` (786 MB fp32). ### 4. Build log `✅ Build complete in 633.6s` (Export 369.6s + Optimize 256.9s; **no quantize** — `quant: null`). ONNX IR 8, opset 17, external data co-located, output `logits[1,1]`. ### 5. Appended findings `model_knowledge/deberta.json` → `deberta-002` (second model in the `deberta` family; `mxbai-rerank-base-v1` added to `models_tested`). Lane A (skill repo); **not** part of this model PR's diff. ### 6. Optimum-coverage probe `deberta-v2` `text-classification` is **VENDOR-ONLY** (`added_by_winml = []`). Vendor onnx tasks: `[feature-extraction, fill-mask, multiple-choice, question-answering, text-classification, token-classification]`. No winml-added exporter exists or is needed. ### 7. Claimed (Effort, Goal, Outcome) **L0★ / L2 / L0.** Baseline: `winml build -m mixedbread-ai/mxbai-rerank-base-v1` on `main` (`origin/main = e95011d` — branch rebased onto current main; winml 0.2.0) already **PASSES** (default pipeline 973.3s incl. quantize → uint8/16 327.5 MB). Contribution = curated CPU float recipe (fp32 + fp16) + L2 relevance-head-preservation proof. ### 8. Goal-ladder verdict table | Tier | Verdict | Evidence | |---|---|---| | L0 (build) | **PASS** | recipe-driven float build exit 0 (633.6 s, no quantize); ONNX opset 17, inputs `input_ids[1,512]`+`attention_mask[1,512]`, output `logits[1,1]`, external data co-located | | L1 (perf) | **PASS** | CPU/fp32 avg 7027.85 ms, 0.14 samples/s, RAM Δ +624.7 MB (also auto-ran on QNN/NPU at 523.99 ms / 1.91 samples/s) | | L2 (numeric vs PyTorch) | **PASS** | 4 real `(query, doc)` pairs: cosine **1.000000**, max-abs **1.097e-05**, **ranking order 4/4 identical**, top-1 doc agrees | Ceiling L2 reached; no downgrade. ### 9. Methodology-evolution declaration No methodology friction. This is the **second** model in the `deberta` family (added in #1117) — it exercised the self-learning path as designed: `deberta-001` predicted the recipe shape, the recipe came out byte-identical (same git blob OID `7b36e695…`), and `deberta-002` records the reranker-specific ranking-parity check. No skill_meta change needed. ### 10. Perf & eval data | EP / Device | Precision | Verdict | Mean | p50 | Throughput | RAM Δ | Task metric | |---|---|---|---|---|---|---|---| | CPUExecutionProvider / cpu | fp32 | PASS | 7027.85 ms | 6933.79 ms | 0.14 samples/s | +624.7 MB | N/A (L3 not marched — L2 ceiling) | | QNNExecutionProvider / npu | (auto) | PASS (bonus) | 523.99 ms | 522.76 ms | 1.91 samples/s | +1791.7 MB | — (not a full NPU validation) | `winml eval` not run (Goal ceiling L2). The NPU row is a bonus data point from `winml perf` auto-selecting QNN — it is **not** a claimed coverage bucket (no NPU-side L2 parity yet), so coverage stays CPU-only/partial. ### 11. Component / op-level data `winml analyze --ep all`: **568 total operators, 17 unique** (identical op profile to the NLI sibling). Per-EP: **QNN NPU 17/17 supported**, **OpenVINO NPU 17/17 supported** (includes the disentangled-attention `GatherElements` ops); **VitisAI all-unknown** (no rule data — analyze exit 1, expected, not a functional failure). Artifact: `temp/mxbai_rerank/analyze_all.json`. ### 12. Reproducible commands ```powershell # baseline (main already builds this model, no recipe) winml build -m mixedbread-ai/mxbai-rerank-base-v1 -o temp\mxbai_rerank_baseline # recipe-driven float build (this PR) winml build -m mixedbread-ai/mxbai-rerank-base-v1 ` -c examples\recipes\mixedbread-ai_mxbai-rerank-base-v1\cpu\cpu\text-classification_fp32_config.json ` -o temp\mxbai_rerank # L1 perf (CPU, pinned) winml perf -m temp\mxbai_rerank\model.onnx --device cpu --ep cpu --iterations 15 --warmup 3 --no-analyze # op coverage winml analyze -m temp\mxbai_rerank\model.onnx --ep all -o temp\mxbai_rerank\analyze_all.json # L2 parity + ranking vs PyTorch: temp\mxbai_rerank_l2.py # (tokenizes (query, doc) cross-encoder pairs, pad to 512, int32 ids, drop token_type_ids, # runs each pair at batch=1 since the recipe fixes batch_size=1, compares logits + argsort) python temp\mxbai_rerank_l2.py ``` Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR adds support for
audeering/wav2vec2-large-robust-12-ft-emotion-msp-dim, routed asaudio-classificationwith an emotion regression head (wav2vec2_emotion_regression). The shipped outcome is L1 on the CPU target EP, and the produced ONNX artifact is fp32 with a matching fp32 recipe name. The highest Goal verdict honestly reached by the tester is L2 PASS, with cosine 0.9999998807907104 and max_abs 1.6689300537109375e-06.Recipe path(s)
examples/recipes/audeering_wav2vec2-large-robust-12-ft-emotion-msp-dim/cpu/cpu/audio-classification_fp32_config.jsonREADME row
false— fp32/CPU-only; not added to the fp16-on-all-buckets tableBuild output dir
temp\fix_build\Build log
✅ Build complete in 256.5sAppended findings
model_knowledge/wav2vec2.jsonfindingswav2vec2-001..wav2vec2-005were appended on the skill repo Lane A, not in this model PR.optim.gelu_fusion=true,optim.matmul_add_fusion=true, quant stayed null.Optimum-coverage probe
VENDOR-ONLYadded_by_winml=[]Claimed (Effort, Goal, Outcome)
["cpu"]130acfe42523b8aa553b1dd10eecd7a1328b832e(currentorigin/mainat rebase time)winml --version:winml, version 0.2.0winml build -c examples\recipes\audeering_wav2vec2-large-robust-12-ft-emotion-msp-dim\cpu\cpu\audio-classification_fp32_config.json -m audeering/wav2vec2-large-robust-12-ft-emotion-msp-dim -o temp\fix_build\; build exit 0;✅ Build complete in 256.5s; ONNX inputinput_values[1,16000], outputshidden_states[1,1024]andlogits[1,3]; initializer dtypes{FLOAT:233, INT64:10}withFLOAT16_COUNT 0, correctly fp32.Goal-ladder verdict table
winml build -c examples\recipes\audeering_wav2vec2-large-robust-12-ft-emotion-msp-dim\cpu\cpu\audio-classification_fp32_config.json -m audeering/wav2vec2-large-robust-12-ft-emotion-msp-dim -o temp\fix_build\winml perf -m temp\fix_build\model.onnx --device cpu --ep cpupython temp\audeering_l2_parity.pyMethodology-evolution declaration
_meta-056effort-mis-estimate (planner Optimum-probe edit),_meta-057goal-ceiling mis-estimate (planner Goal-axis edit), and_meta-058doc-code-driftuser_skill→dev_skill(explainer/reviewer path fix) were filed on the SKILL repo Lane A branch.9d380baeand are intentionally not included in this model PR to keep lanes unpolluted.Perf & eval data
winml analyze --model temp\fix_build\model.onnx --ep all --format jsontemp\fix_analyze_output.txt