Skip to content

Add LayoutLM v1 document-question-answering support (impira/layoutlm-invoices)#1096

Draft
ssss141414 wants to merge 5 commits into
mainfrom
shzhen/add-layoutlm-invoices
Draft

Add LayoutLM v1 document-question-answering support (impira/layoutlm-invoices)#1096
ssss141414 wants to merge 5 commits into
mainfrom
shzhen/add-layoutlm-invoices

Conversation

@ssss141414

@ssss141414 ssss141414 commented Jul 12, 2026

Copy link
Copy Markdown
Contributor

Add LayoutLM v1 document-question-answering support — impira/layoutlm-invoices

Lane-A PR. Adds:

  • src/winml/modelkit/models/hf/layoutlm.pyLayoutLMDocumentQAOnnxConfig (registers a document-question-answering OnnxConfig for layoutlm, overriding Optimum's single-logits default with the extractive-QA start_logits/end_logits pair; adjusts max_position_embeddings for the RoBERTa-style pad-token offset so ONNX tracing does not hit a position index out-of-bounds)
  • registration in src/winml/modelkit/models/hf/__init__.py
  • recipe examples/recipes/impira_layoutlm-invoices/cpu/cpu/document-question-answering_fp16_config.json

Environment (this run)

  • winml-cli v0.2.0 (editable install)
  • Base: rebased onto origin/main 3f5e4683 — branch HEAD 175b8e76
  • Host EPs (onnxruntime.get_available_providers()): CPUExecutionProvider, AzureExecutionProvider → only CPU is runtime-verified this run. Accelerator buckets are carried-over / prior-verified on capable hardware.

Why this is needed (baseline gap — proven)

Clean origin/main, without the new OnnxConfig:

winml build -m impira/layoutlm-invoices
→ FAIL: Unrecognized configuration class LayoutLMConfig for
        AutoModelForNextSentencePrediction.

Optimum registers layoutlm only for feature-extraction / fill-mask / text-/token-classification (no document-QA), and auto task detection mis-routes to a next-sentence head. The new config + recipe (task = document-question-answering, model_class = AutoModelForDocumentQuestionAnswering) route to the correct extractive-QA head and emit start_logits / end_logits.

fp16 caveat (documented)

Recipe quant.mode = "fp16" alone is not honored on CPU (auto-precision cpu→fp32). Pass --precision fp16:

winml build -m <model> -c <recipe> --precision fp16

→ correct fp16 (model.onnx.data ≈ 242.6 MB, half of fp32's 485.2 MB).

Input constraints (for perf / numeric — documented)

Random inputs crash this model; valid inputs are required:

  • token_type_ids must be 0 (this RoBERTa-vocab checkpoint has type_vocab_size = 1; index 1 is out of bounds in the token-type embedding).
  • bbox must be geometrically valid (x1 ≥ x0, y1 ≥ y0) so LayoutLM's height/width position embeddings receive non-negative indices. PyTorch nn.Embedding raises on negative indices (ORT Gather silently wraps).

Goal ladder (CPU, this run)

Level Check Result
L0 build (CPU, recipe, --precision fp16) PASS — 98.1 s (export 19.6 s + optimize 44.6 s); fp16 242.6 MB
L1 perf (CPU, fp16, constrained inputs) PASS — P50 269.27 ms, P90 279.81 ms, avg 269.08 ms, tput 3.72/s; RAM load +346 MB, inference +125.5 MB; inputs input_ids/attention_mask/token_type_ids[1,512] + bbox[1,512,4] int32, outputs start_logits[1,512] / end_logits[1,512]
L2 numeric parity vs PyTorch AutoModelForDocumentQuestionAnswering PASS — fp32 (optimized.onnx) start max_abs 1.05e-05 / end 1.14e-05, cosine 1.000000; fp16 (model.onnx) start 1.82e-02 / end 2.01e-02, cosine 1.000000 (fp16 tolerance)
L3 accuracy eval Under-reached — extractive doc-QA; no default DocVQA-style dataset in the eval harness
analyze op-level --ep all OpenVINO(CPU) 397/0/0/0 (all supported, ready to deploy); CPU EP no rule data

Numeric parity is clean. The new OnnxConfig correctly exports the extractive-QA head and the recipe fixes task/model-class detection.

…invoices)

Register a document-question-answering OnnxConfig for layoutlm (Optimum has none for v1) that emits extractive-QA outputs (start_logits/end_logits) and applies the RoBERTa-style position-offset adjustment used by this checkpoint. Add an fp16 recipe with corrected sequence length (512) and embedding-safe input value ranges (token_type_ids pinned to 0 for type_vocab_size=1). Validated fp16 build + CPU inference (P50 280.6ms).
@ssss141414 ssss141414 force-pushed the shzhen/add-layoutlm-invoices branch from 37278b0 to 175b8e7 Compare July 15, 2026 04:13
@ssss141414 ssss141414 added the model-scale-by-skill Model support PR created or maintained by the adding-model-support skill label Jul 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

model-scale-by-skill Model support PR created or maintained by the adding-model-support skill

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant