Skip to content

feat(florence2): add Florence-2 image-to-text support#1098

Draft
timenick wants to merge 5 commits into
mainfrom
timenick-zhiwang-add-florence-2-support
Draft

feat(florence2): add Florence-2 image-to-text support#1098
timenick wants to merge 5 commits into
mainfrom
timenick-zhiwang-add-florence-2-support

Conversation

@timenick

Copy link
Copy Markdown
Collaborator

Adds CPU FP16 image-to-text support for microsoft/Florence-2-base in WinML CLI using split encoder/decoder exports and the <CAPTION> prompt. Claimed tier: Effort L2, Goal L3, Outcome L2. Goal L2 passed; Goal L3 is HOST-BLOCKED because pinned Flickr30K local materialization repeatedly raised WinError 448.

  1. Recipe paths
    • examples/recipes/microsoft_Florence-2-base/image-to-text_fp16_config_encoder.json
    • examples/recipes/microsoft_Florence-2-base/image-to-text_fp16_config_decoder.json
  2. README rowmicrosoft/Florence-2-base | image-to-text
  3. Build output directory — ephemeral model-specific temp paths; cleaned after evidence capture.
  4. Build result — both CPU FP16 components built successfully; encoder FLOAT16 514/551, decoder 164/179.
  5. Appended findings — skill-only model_knowledge/florence2.json, findings -001 through -009; excluded from this PR.
  6. Optimum coverage — WINML-ONLY composite support.
  7. Claimed tier — Effort L2; Goal L3; Outcome L2; CPU FP16 <CAPTION>.
  8. Goal ladder
    • L0 PASS: actual FP16 encoder/decoder and assembled generation.
    • L1 PASS: encoder 528.355ms; decoder 9.405ms; composite first 487.519ms/cached 18.085ms.
    • L2 PASS: encoder cosine/maxabs .999999999447/.0315857; decoder first .999999999999891/.0000191; cached .999999999999870/.00000954; assembled .999999999999870/.00001812.
    • L3 HOST-BLOCKED: pinned Flickr30K materialization repeatedly raised WinError 448.
  9. Methodology evolution — no separate methodology finding.
  10. Perf/eval — CPU FP16 L1 passed; L3 metric deferred as HOST-BLOCKED.
  11. Component data — split encoder/decoder artifacts validated independently and as an assembled pipeline.
  12. Reproducible build commands
uv run winml build -c examples\recipes\microsoft_Florence-2-base\image-to-text_fp16_config_encoder.json -m temp\microsoft_Florence-2-base\model --device cpu --precision fp16 --trust-remote-code --no-compile -o temp\microsoft_Florence-2-base\encoder
uv run winml build -c examples\recipes\microsoft_Florence-2-base\image-to-text_fp16_config_decoder.json -m temp\microsoft_Florence-2-base\model --device cpu --precision fp16 --trust-remote-code --no-compile -o temp\microsoft_Florence-2-base\decoder

Validation notes: 297/297 changed unit-test items reached 100%; pytest session cleanup then hit host WinError 448. Florence integration passed 4/4. Full tests/ was interrupted by an unrelated OpenVINO integration timeout. Changed-file Ruff and git diff --check passed. Independent review found no remaining Critical/Important issues.

timenick added 2 commits July 13, 2026 09:16
Add Florence-2 split encoder/decoder export support, CPU FP16 recipes, and composite image-to-text integration.

Extend shared composite processing, inspection, evaluation, shape, and configuration paths with targeted coverage. CPU FP16 passes build, perf, and numeric validation; Flickr30K task evaluation remains HOST-BLOCKED by WinError 448 during local materialization.
Comment thread tests/unit/onnx/test_shape_inference.py Fixed
@ssss141414 ssss141414 added the model-scale-by-skill Model support PR created or maintained by the adding-model-support skill label Jul 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

model-scale-by-skill Model support PR created or maintained by the adding-model-support skill

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants