Add ASR fp16 recipe for MahmoudAshraf/mms-300m-1130-forced-aligner

[ssss141414]

MahmoudAshraf/mms-300m-1130-forced-aligner: generalized CTC loader fix and fp16 recipe

This PR fixes ASR model-class resolution for CTC architectures and adds the validated CPU/fp16 recipe for MahmoudAshraf/mms-300m-1130-forced-aligner.

Root cause

The automatic-speech-recognition task covers two incompatible model families:

• encoder-only CTC models, loaded by AutoModelForCTC;
• encoder-decoder speech models, loaded by AutoModelForSpeechSeq2Seq.

Before this fix, task-only resolution selected AutoModelForSpeechSeq2Seq for this published Wav2Vec2ForCTC checkpoint and failed with:

Unrecognized configuration class Wav2Vec2Config for AutoModelForSpeechSeq2Seq

Generalized code fix

Model-class resolution now uses the checkpoint's published architecture metadata when the task is ambiguous:

• ASR architectures ending in ForCTC resolve to AutoModelForCTC;
• both automatic-task and explicit-task paths use the same rule;
• no model ID is hardcoded;
• seq2seq speech models retain the existing fallback behavior.

A regression test confirms WhisperForConditionalGeneration still resolves to AutoModelForSpeechSeq2Seq.

The recipe no longer sets loader.model_class; it relies on the generalized resolver. It now supplies only the task/model type plus fp16 export and EP settings.

Recipe

Added:

examples/recipes/MahmoudAshraf_mms-300m-1130-forced-aligner/cpu/cpu/automatic-speech-recognition_fp16_config.json

On CPU, pass --precision fp16 explicitly because automatic CPU precision otherwise selects fp32.

Validation

• No-recipe inspect/config: Wav2Vec2ForCTC -> AutoModelForCTC.
• No-recipe CPU build: PASS, 137.8 s (export 92.8 s, optimize 42.6 s).
• Input/output: input_values[1,16000] -> logits.
• Unit tests: 17 passed in tests/unit/loader/test_config_resolution.py.
• Covered paths: CTC auto-task, CTC explicit-task, and Whisper seq2seq regression.
• Ruff: PASS.

fp16 goal-ladder evidence

Level	Check	Result
L0	CPU build with recipe and --precision fp16	PASS, 216.8 s, external data about 601.7 MB
L1	CPU perf	PASS, P50 126.51 ms, P90 136.67 ms, 7.88 samples/s
L2	Numeric parity	PASS, fp32 cosine 1.000000; fp16 cosine 0.999999
L3	Forced-alignment accuracy	Not available in the current evaluation harness

The historical baseline failure is now closed by reusable code; the recipe is retained for fp16 and EP configuration rather than as a model-class workaround.

[xieofxie]

if without the config, could perf run it directly?

[ssss141414]

if without the config, could perf run it directly?

As PR's description, it will failed on build stage without config. And the following command will not be run (perf/ eval)...

This is a standard Wav2Vec2ForCTC model (architectures=["Wav2Vec2ForCTC"], vocab_size=31), not trust_remote_code. But winml build auto-config resolves automatic-speech-recognition to a seq2seq loader and fails:

Error: Unrecognized configuration class Wav2Vec2Config for this kind of
AutoModel: AutoModelForSpeechSeq2Seq. Model type should be one of DiaConfig,
... SpeechEncoderDecoderConfig, ... WhisperConfig.

[xieofxie]

if without the config, could perf run it directly?

As PR's description, it will failed on build stage without config. And the following command will not be run (perf/ eval)...

This is a standard Wav2Vec2ForCTC model (architectures=["Wav2Vec2ForCTC"], vocab_size=31), not trust_remote_code. But winml build auto-config resolves automatic-speech-recognition to a seq2seq loader and fails:

Error: Unrecognized configuration class Wav2Vec2Config for this kind of
AutoModel: AutoModelForSpeechSeq2Seq. Model type should be one of DiaConfig,
... SpeechEncoderDecoderConfig, ... WhisperConfig.

if it is the case, should we update the code to make it work?

should fix model class mapping?

[ssss141414]

Updated in 0976f77. The code now disambiguates ASR loaders from published architecture metadata: *ForCTC resolves to AutoModelForCTC in both auto-task and explicit-task paths, without a model-ID allowlist. Whisper remains on AutoModelForSpeechSeq2Seq. The real MMS model now completes a no-recipe build in 137.8 s; 17 loader-resolution tests and Ruff pass. The recipe is retained only for fp16/EP settings, not as a required model-class workaround.

[xieofxie]

Yes — this makes sense. The diagnosis is correct: automatic-speech-recognition is an ambiguous pipeline task, and task-only resolution picks AutoModelForSpeechSeq2Seq, which can't load a Wav2Vec2Config CTC checkpoint. Resolving from the published architectures metadata is the right layer to disambiguate, and the fix is properly architecture-driven with no model-id hardcoding (satisfies the repo's no-hardcoded-logic rule).

Verified locally at 7e972a0:

• ruff check clean;
• tests/unit/loader/test_config_resolution.py → 17 passed (CTC auto-task, CTC explicit-task, Whisper seq2seq regression);
• adjacent resolver suites (test_detect_task, test_detect_task_and_class, test_resolve_task_override, test_known_tasks) → 90 passed, no regressions.

The override is correctly inserted before the TasksManager default in both the user-task and detection paths, so custom MODEL_CLASS_MAPPING entries still win and seq2seq speech models keep AutoModelForSpeechSeq2Seq. Returning the AutoModelForCTC auto-class (rather than the concrete Wav2Vec2ForCTC) keeps the two ASR families symmetric — good call. The recipe correctly drops loader.model_class now that resolution is automatic, follows the <model>/<ep>/<device>/<task>_<precision>_config.json layout, and its quant block is just the standard to_dict serialization (the int8/calibration fields are inert for mode="fp16").

One design-discussion note inline; non-blocking. Also a doc nit on the PR description below.

[xieofxie]

Design question (non-blocking): this introduces a second architecture-based resolution mechanism alongside the existing _resolve_model_class_from_config, which already reads architectures[0] and would return the concrete Wav2Vec2ForCTC. That helper only runs as the last fallback in Stage 2 (after TasksManager succeeds with the wrong seq2seq class), which is exactly why the bug slips through — so inserting an arch check before TasksManager is the correct move.

The trade-off worth calling out: this is a hand-maintained (task, suffix) allowlist, so every future ambiguous-task family (e.g. another dual-family pipeline task) needs a new entry. An alternative would be to prefer the concrete config architecture over the TasksManager default whenever architectures is present and the task is known-ambiguous — generalizing without a suffix table. I think the targeted map is the safer, more reviewable choice here (it can't regress the many tasks where the Auto* class is intentionally preferred), so I'd keep it — just flagging that the map will accrete entries over time. A short comment noting "add an entry per ambiguous task family" on _ARCHITECTURE_AUTO_CLASS_OVERRIDES would help the next person.