Fix test_mnb_to_qdq for ORT versions that fixed the int4→uint8 DQ bug

[Copilot]

Describe your changes

test_mnb_to_qdq hard-asserted that ORT raises a "uint8" exception for the symmetric + signed_int + no_zero_point + transpose_op case, a workaround for an old ORT graph-optimization bug that rewrote int4 DQ nodes to uint8. Newer ORT builds fixed that bug, so the session now loads successfully and pytest.raises(Exception, match="uint8") fails with DID NOT RAISE.

• Version-tolerant handling: Replaced the strict pytest.raises with a try/except that skips remaining validation only if the legacy "uint8" error still occurs (old ORT), re-raises anything unexpected, and otherwise proceeds to the normal output-correctness checks (fixed ORT).

Test-only change; no production code affected.

Checklist before requesting a review

• Add unit tests for this change.
• Make sure all tests can pass.
• Update documents if necessary.
• Lint and apply fixes to your code by running lintrunner -a
• Is this a user-facing change? If yes, give a description of this change to be included in the release notes.

(Optional) Issue link

• Fixes CI stuck, test_mnb_to_qdq is failing #2594

[azure-pipelines]

Azure Pipelines:
There may be pipelines that require an authorized user to comment /azp run to run.

[xadupre]

@copilot i think the latest version of onnxruntime does not crash anymore so this test should conditioned to the fact onnxruntime version is older than a specific reason

[Copilot]

@copilot i think the latest version of onnxruntime does not crash anymore so this test should conditioned to the fact onnxruntime version is older than a specific reason

Done in 2d? — the uint8 crash is now only expected when onnxruntime < 1.28.0 (via a new INT4_DQ_UINT8_BUG version check). CI runs 1.28.0, where the bug is fixed, so it now runs the normal output-correctness validation instead of asserting the crash.

@copilot i think the latest version of onnxruntime does not crash anymore so this test should conditioned to the fact onnxruntime version is older than a specific reason

Correction on the commit reference above: this is addressed in 59cdbf9.

[Copilot]

Pull request overview

This PR updates Olive’s ONNX pass test suite to handle an ONNX Runtime (ORT) behavior change: newer ORT builds no longer hit the legacy int4→uint8 dequantization rewrite that previously caused test_mnb_to_qdq to fail while loading the generated model.

Changes:

• Added an ORT-version flag (INT4_DQ_UINT8_BUG) documenting and gating the legacy int4 DQ → uint8 DQ optimization issue.
• Updated test_mnb_to_qdq to only expect the "uint8" load failure on ORT versions believed to still have the bug, allowing newer ORT versions to proceed to output validation.

[xadupre]

@copilot fix lint issues and fast tests

[Copilot]

@copilot fix lint issues and fast tests

Fixed in a905020. I reduced the boolean-heavy condition to satisfy pylint and re-ran the targeted fast test for this file (python -m pytest -q test/passes/onnx/test_mnb_to_qdq.py), which passed.

[xadupre]

@copilot SUBFAILED(model_id='local/tiny-random-llama-a') test/cli/test_cli_test_model_smoke.py::TestCliTestModelSmoke::test_documented_test_model_smoke_flow - AssertionError: assert False

• where False = <built-in method issubset of set object at 0x7f146ef20900>({'olive_test_output.json', 'reference_hf_model/config.json', 'reference_hf_model/generation_config.json', 'reference_hf_model/model.safetensors', 'reference_hf_model/olive_test_model.json', 'reference_hf_model/tokenizer.json', ...})
• where <built-in method issubset of set object at 0x7f146ef20900> = {'config.json', 'genai_config.json', 'generation_config.json', 'model.onnx', 'model_config.json', 'olive_test_output.json', ...}.issubset
  SUBFAILED(model_id='local/tiny-random-llama-b') test/cli/test_cli_test_model_smoke.py::TestCliTestModelSmoke::test_documented_test_model_smoke_flow - AssertionError: assert False
• where False = <built-in method issubset of set object at 0x7f146ef20900>({'olive_test_output.json', 'reference_hf_model/config.json', 'reference_hf_model/generation_config.json', 'reference_hf_model/model.safetensors', 'reference_hf_model/olive_test_model.json', 'reference_hf_model/tokenizer.json', ...})
• where <built-in method issubset of set object at 0x7f146ef20900> = {'config.json', 'genai_config.json', 'generation_config.json', 'model.onnx', 'model_config.json', 'olive_test_output.json', ...}.issubset
  SUBFAILED(model_id='mistralai/Mistral-7B-Instruct-v0.3') test/cli/test_cli_test_model_smoke.py::TestCliTestModelSmoke::test_documented_test_model_smoke_flow - AssertionError: assert False
• where False = <built-in method issubset of set object at 0x7f146ef20900>({'olive_test_output.json', 'reference_hf_model/config.json', 'reference_hf_model/generation_config.json', 'reference_hf_model/model.safetensors', 'reference_hf_model/olive_test_model.json', 'reference_hf_model/tokenizer.json', ...})
• where <built-in method issubset of set object at 0x7f146ef20900> = {'config.json', 'genai_config.json', 'generation_config.json', 'model.onnx', 'model_config.json', 'olive_test_output.json', ...}.issubset
  SUBFAILED(model_id='microsoft/Phi-3-mini-4k-instruct') test/cli/test_cli_test_model_smoke.py::TestCliTestModelSmoke::test_documented_test_model_smoke_flow - AssertionError: assert False
• where False = <built-in method issubset of set object at 0x7f146ef20900>({'olive_test_output.json', 'reference_hf_model/config.json', 'reference_hf_model/generation_config.json', 'reference_hf_model/model.safetensors', 'reference_hf_model/olive_test_model.json', 'reference_hf_model/tokenizer.json', ...})
• where <built-in method issubset of set object at 0x7f146ef20900> = {'config.json', 'genai_config.json', 'generation_config.json', 'model.onnx', 'model_config.json', 'olive_test_output.json', ...}.issubset
  SUBFAILED(model_id='Qwen/Qwen3-8B') test/cli/test_cli_test_model_smoke.py::TestCliTestModelSmoke::test_documented_test_model_smoke_flow - AssertionError: assert False
• where False = <built-in method issubset of set object at 0x7f146ef20900>({'olive_test_output.json', 'reference_hf_model/config.json', 'reference_hf_model/generation_config.json', 'reference_hf_model/model.safetensors', 'reference_hf_model/olive_test_model.json', 'reference_hf_model/tokenizer.json', ...})
• where <built-in method issubset of set object at 0x7f146ef20900> = {'config.json', 'genai_config.json', 'generation_config.json', 'model.onnx', 'model_config.json', 'olive_test_output.json', ...}.issubset
  SUBFAILED(model_id='openai/whisper-tiny') test/cli/test_cli_whisper_smoke.py::TestCliWhisperSmoke::test_whisper_capture_onnx_graph - AssertionError: Expected encoder.onnx in output, got: set()
  assert 'encoder.onnx' in set()
  SUBFAILED(model_id='microsoft/whisper-base') test/cli/test_cli_whisper_smoke.py::TestCliWhisperSmoke::test_whisper_capture_onnx_graph - AssertionError: Expected encoder.onnx in output, got: set()
  assert 'encoder.onnx' in set()

[Copilot]

@copilot SUBFAILED(model_id='local/tiny-random-llama-a') test/cli/test_cli_test_model_smoke.py::TestCliTestModelSmoke::test_documented_test_model_smoke_flow - AssertionError: assert False

• where F...

Fixed in 55ed28e. The fast-test failure was caused by two runtime compatibility gaps: the model builder now needs parse_extra_options(...) before create_model(...), and the discrepancy check needed to infer kv_cache_dim from the exported model config. I re-ran the targeted smoke test (python -m pytest -q test/cli/test_cli_test_model_smoke.py -k documented_test_model_smoke_flow), and it passed.

[xadupre]

@copilot _ TestCliTestModelSmoke.test_model_discrepancy (model_id='Qwen/Qwen2.5-VL-7B-Instruct', exporter='model_builder') _
test/cli/test_cli_test_model_smoke.py:395: in _assert_discrepancy
self._assert_discrepancy_model_builder(tmp_path, model_id)
test/cli/test_cli_test_model_smoke.py:447: in _assert_discrepancy_model_builder
self._run_discrepancy_with_test(config_path, run_output_dir)
test/cli/test_cli_test_model_smoke.py:405: in _run_discrepancy_with_test
_run_cli_main(
test/cli/test_cli_test_model_smoke.py:250: in _run_cli_main
cli_main(args)
olive/cli/launcher.py:79: in main
service.run()
olive/telemetry/telemetry_extensions.py:137: in wrapper
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
olive/cli/run.py:89: in run
workflow_output = olive_run(
olive/workflows/run/run.py:178: in run
return run_engine(package_config, run_config)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
olive/workflows/run/run.py:139: in run_engine
return engine.run(
olive/telemetry/telemetry_extensions.py:137: in wrapper
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
olive/engine/engine.py:210: in run
self.run_accelerator(
olive/engine/engine.py:285: in run_accelerator
self._run_no_search(input_model_config, input_model_id, accelerator_spec, artifacts_dir)
olive/engine/engine.py:329: in _run_no_search
should_prune, signal, model_ids = self._run_passes(input_model_config, input_model_id, accelerator_spec)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
olive/engine/engine.py:628: in _run_passes
model_config, model_id = self._run_pass(
olive/engine/engine.py:722: in _run_pass
output_model_config = host.run_pass(p, input_model_config, output_model_path)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
olive/systems/local.py:45: in run_pass
output_model = the_pass.run(model, output_model_path)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
olive/passes/olive_pass.py:243: in run
output_model = self._run_for_config(model, self.config, output_model_path)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
olive/passes/onnx/discrepancy_check.py:457: in _run_for_config
dataloader, io_config = self._prepare_dataloader(model)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
olive/passes/onnx/discrepancy_check.py:563: in _prepare_dataloader
new_shape = _infer_shape(shape, known)
^^^^^^^^^^^^^^^^^^^^^^^^^^
olive/passes/onnx/discrepancy_check.py:75: in _infer_shape
f"Known symbols are: {sorted(default_values)}. "
^^^^^^^^^^^^^^^^^^^^^^
E TypeError: '<' not supported between instances of 'int' and 'str'
=========================== short test summary info ============================
SUBFAILED(model_id='Qwen/Qwen2.5-VL-7B-Instruct', exporter='model_builder') test/cli/test_cli_test_model_smoke.py::TestCliTestModelSmoke::test_model_discrepancy - TypeError: '<' not supported between instances of 'int' and 'str'

[xadupre]

fixed in another PR