Skip to content

Fix multi-input ONNX binding and add OWLv2 CPU fp32 recipe#1101

Open
ssss141414 wants to merge 6 commits into
mainfrom
shzhen/add-owlv2-base-patch16
Open

Fix multi-input ONNX binding and add OWLv2 CPU fp32 recipe#1101
ssss141414 wants to merge 6 commits into
mainfrom
shzhen/add-owlv2-base-patch16

Conversation

@ssss141414

@ssss141414 ssss141414 commented Jul 13, 2026

Copy link
Copy Markdown
Contributor

Add OWLv2 zero-shot object detection support (CPU fp32)

Adds verified CPU fp32 coverage for google/owlv2-base-patch16 and fixes the generalized multi-input ONNX export bug discovered while producing the recipe.

Why this is not recipe-only

The initial recipe listed [pixel_values, input_ids, attention_mask]. torch.onnx.export invoked the model safely with keyword arguments, but applied input_names positionally in forward() order. For OWLv2 (input_ids, pixel_values, attention_mask), the ONNX graph therefore labeled the rank-2 text tensor as pixel_values and the rank-4 image tensor as input_ids. Named-input perf/eval then failed with Invalid rank for input: pixel_values Got: 4 Expected: 2.

Reordering one recipe only hid this framework bug. Commit 72cd79ef fixes HTPExporter._convert_model_to_onnx() so the default kwargs export path derives ONNX input-name order from the model's forward() signature. Models implementing the existing get_export_args() positional protocol retain their explicit binding behavior.

The existing regression test previously asserted only that both names existed, so swapped bindings passed. It now asserts the exact name-to-shape mapping and reproduces the former failure when run against old main.

Recipe

  • Path: examples/recipes/google_owlv2-base-patch16/cpu/cpu/zero-shot-object-detection_fp32_config.json
  • Inputs: input_ids [1,16] int32, pixel_values [1,3,960,960] float32, attention_mask [1,16] int32
  • Outputs: logits, pred_boxes, text_embeds, image_embeds
  • Optimization: clamp_constant_values=true
  • Precision: CPU fp32 (quant: null)

The earlier _fp16 filename was inaccurate: the recipe uses quant: null and this run contains only fp32 evidence. Commit 920a3047 renames it to _fp32. No fp16 or accelerator recipe is claimed without exact tuple evidence.

Verification

Level Result
Exporter unit tests PASS ? 29 tests (test_pytorch_export.py plus related patcher tests)
Ruff PASS ? exporter and regression test
L0 build PASS ? CPU fp32; model.onnx.data = 585.2 MB
L1 perf PASS ? CPU fp32; avg 3959.46 ms, P50 3963.58 ms, P90 4065.66 ms
L2 numeric PASS ? all four outputs match PyTorch; cosine 1.000000
L3 accuracy Not reached ? no supported ZSOD dataset/evaluator in the current harness

Precision / EP evidence matrix

EP Device Precision Verdict
CPU CPU fp32 PASS ? recipe included
CPU CPU fp16 NOT VERIFIED ? no recipe included
Other CPU/GPU EPs CPU/GPU fp32, fp16 HOST-BLOCKED / not verified ? no recipes included
NPU EPs NPU fp32, fp16, w8a8, w8a16 HOST-BLOCKED / not verified ? no recipes included

winml analyze --ep all on the available host reported OpenVINO CPU 1020 supported / 5 unknown boundary casts. Provider availability is not used to infer support for untested tuples.

@ssss141414 ssss141414 closed this Jul 13, 2026
@ssss141414 ssss141414 reopened this Jul 15, 2026
@ssss141414 ssss141414 force-pushed the shzhen/add-owlv2-base-patch16 branch from 6b10ded to df5aee3 Compare July 15, 2026 06:03
@ssss141414 ssss141414 marked this pull request as ready for review July 15, 2026 08:32
@ssss141414 ssss141414 requested a review from a team as a code owner July 15, 2026 08:32
@ssss141414 ssss141414 changed the title Add google/owlv2-base-patch16 zero-shot-object-detection fp16 recipe Fix multi-input ONNX binding and add OWLv2 CPU fp32 recipe Jul 16, 2026
@ssss141414 ssss141414 added the model-scale-by-skill Model support PR created or maintained by the adding-model-support skill label Jul 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

model-scale-by-skill Model support PR created or maintained by the adding-model-support skill

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant