Skip to content

feat(florence2): add Florence-2 image-to-text support #1098

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
timenick wants to merge 5 commits into
base: main
Choose a base branch
from
Draft

feat(florence2): add Florence-2 image-to-text support #1098

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
04dac65
feat(florence2): add Florence-2 image-to-text support
timenick Jul 13, 2026
1801369
Merge remote-tracking branch 'origin/main' into timenick-zhiwang-add-…
timenick Jul 13, 2026
38a1f29
fix(florence2): satisfy type checking
timenick Jul 13, 2026
8949c2e
test(onnx): create shape inference temp parents
timenick Jul 13, 2026
fc976a1
test(onnx): use consistent import style
timenick Jul 13, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions docs/reference/index.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -67,6 +67,7 @@ stages based on the target device and precision.
| `dtype` | `str \| null` | Data type (e.g., `float32`, `int64`). |
| `shape` | `list[int \| str] \| null` | Tensor shape (e.g., `[1, 3, 224, 224]`). String entries declare symbolic dynamic axes and use size `1` for dummy inputs. |
| `value_range` | `[float, float] \| null` | Min/max for dummy tensor generation. |
| `dummy_value_runs` | `list[[int, int \| float]] \| null` | Run-length encoded deterministic dummy values. Generated configs use this only when it compactly preserves semantic input values; runs must fill the concrete tensor shape exactly. |

---

Expand Down
3 changes: 2 additions & 1 deletion examples/recipes/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ Each *(model, task)* includes:

## Models

Total: **75** (model, task) tuples that pass fp16 eval on all 10 (EP, device) buckets.
Total: **76** (model, task) tuples that pass fp16 eval on all 10 (EP, device) buckets.

| Model | Task |
|---|---|
Expand Down Expand Up @@ -64,6 +64,7 @@ Total: **75** (model, task) tuples that pass fp16 eval on all 10 (EP, device) bu
| joeddav/xlm-roberta-large-xnli | zero-shot-classification |
| laion/CLIP-ViT-B-32-laion2B-s34B-b79K | feature-extraction |
| mattmdjaga/segformer_b2_clothes | image-segmentation |
| microsoft/Florence-2-base | image-to-text |
| microsoft/rad-dino | image-feature-extraction |
| microsoft/resnet-18 | image-classification |
| microsoft/resnet-50 | image-classification |
Expand Down
289 changes: 289 additions & 0 deletions examples/recipes/microsoft_Florence-2-base/image-to-text_fp16_config_decoder.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,289 @@
{
"export": {
"opset_version": 17,
"batch_size": 1,
"export_params": true,
"do_constant_folding": true,
"verbose": false,
"dynamo": false,
"enable_hierarchy_tags": true,
"clean_onnx": false,
"hierarchy_tag_format": "full",
"input_tensors": [
{
"name": "decoder_input_ids",
"dtype": "int32",
"shape": [
1,
1
],
"value_range": [
0,
51289
]
},
{
"name": "encoder_hidden_states",
"dtype": "float32",
"shape": [
1,
16,
768
],
"value_range": [
0,
1
]
},
{
"name": "decoder_attention_mask",
"dtype": "int64",
"shape": [
1,
1024
],
"dummy_value_runs": [
[
1024,
1
]
]
},
{
"name": "cache_position",
"dtype": "int64",
"shape": [
1
]
},
{
"name": "past_0_key",
"dtype": "float32",
"shape": [
1,
12,
1024,
64
],
"value_range": [
0,
1
]
},
{
"name": "past_0_value",
"dtype": "float32",
"shape": [
1,
12,
1024,
64
],
"value_range": [
0,
1
]
},
{
"name": "past_1_key",
"dtype": "float32",
"shape": [
1,
12,
1024,
64
],
"value_range": [
0,
1
]
},
{
"name": "past_1_value",
"dtype": "float32",
"shape": [
1,
12,
1024,
64
],
"value_range": [
0,
1
]
},
{
"name": "past_2_key",
"dtype": "float32",
"shape": [
1,
12,
1024,
64
],
"value_range": [
0,
1
]
},
{
"name": "past_2_value",
"dtype": "float32",
"shape": [
1,
12,
1024,
64
],
"value_range": [
0,
1
]
},
{
"name": "past_3_key",
"dtype": "float32",
"shape": [
1,
12,
1024,
64
],
"value_range": [
0,
1
]
},
{
"name": "past_3_value",
"dtype": "float32",
"shape": [
1,
12,
1024,
64
],
"value_range": [
0,
1
]
},
{
"name": "past_4_key",
"dtype": "float32",
"shape": [
1,
12,
1024,
64
],
"value_range": [
0,
1
]
},
{
"name": "past_4_value",
"dtype": "float32",
"shape": [
1,
12,
1024,
64
],
"value_range": [
0,
1
]
},
{
"name": "past_5_key",
"dtype": "float32",
"shape": [
1,
12,
1024,
64
],
"value_range": [
0,
1
]
},
{
"name": "past_5_value",
"dtype": "float32",
"shape": [
1,
12,
1024,
64
],
"value_range": [
0,
1
]
}
],
"output_tensors": [
{
"name": "logits"
},
{
"name": "present_0_key"
},
{
"name": "present_0_value"
},
{
"name": "present_1_key"
},
{
"name": "present_1_value"
},
{
"name": "present_2_key"
},
{
"name": "present_2_value"
},
{
"name": "present_3_key"
},
{
"name": "present_3_value"
},
{
"name": "present_4_key"
},
{
"name": "present_4_value"
},
{
"name": "present_5_key"
},
{
"name": "present_5_value"
}
],
"dynamic_axes": {
"encoder_hidden_states": {
"0": "batch_size",
"1": "source_sequence_length"
}
}
},
"optim": {
"gelu_fusion": true,
"layer_norm_fusion": true,
"matmul_add_fusion": true
},
"quant": null,
"compile": null,
"loader": {
"task": "text2text-generation",
"model_class": "Florence2DecoderWrapper",
"model_type": "florence2",
"trust_remote_code": true
}
}
Loading
Loading