Add LayoutLM v1 document-question-answering support (impira/layoutlm-invoices)#1096
Draft
ssss141414 wants to merge 5 commits into
Draft
Add LayoutLM v1 document-question-answering support (impira/layoutlm-invoices)#1096ssss141414 wants to merge 5 commits into
ssss141414 wants to merge 5 commits into
Conversation
…invoices) Register a document-question-answering OnnxConfig for layoutlm (Optimum has none for v1) that emits extractive-QA outputs (start_logits/end_logits) and applies the RoBERTa-style position-offset adjustment used by this checkpoint. Add an fp16 recipe with corrected sequence length (512) and embedding-safe input value ranges (token_type_ids pinned to 0 for type_vocab_size=1). Validated fp16 build + CPU inference (P50 280.6ms).
37278b0 to
175b8e7
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add LayoutLM v1 document-question-answering support — impira/layoutlm-invoices
Lane-A PR. Adds:
src/winml/modelkit/models/hf/layoutlm.py—LayoutLMDocumentQAOnnxConfig(registers adocument-question-answeringOnnxConfig forlayoutlm, overriding Optimum's single-logitsdefault with the extractive-QAstart_logits/end_logitspair; adjustsmax_position_embeddingsfor the RoBERTa-style pad-token offset so ONNX tracing does not hit a position index out-of-bounds)src/winml/modelkit/models/hf/__init__.pyexamples/recipes/impira_layoutlm-invoices/cpu/cpu/document-question-answering_fp16_config.jsonEnvironment (this run)
origin/main3f5e4683— branch HEAD175b8e76onnxruntime.get_available_providers()):CPUExecutionProvider,AzureExecutionProvider→ only CPU is runtime-verified this run. Accelerator buckets are carried-over / prior-verified on capable hardware.Why this is needed (baseline gap — proven)
Clean
origin/main, without the new OnnxConfig:Optimum registers
layoutlmonly for feature-extraction / fill-mask / text-/token-classification (no document-QA), and auto task detection mis-routes to a next-sentence head. The new config + recipe (task = document-question-answering,model_class = AutoModelForDocumentQuestionAnswering) route to the correct extractive-QA head and emitstart_logits/end_logits.fp16 caveat (documented)
Recipe
quant.mode = "fp16"alone is not honored on CPU (auto-precision cpu→fp32). Pass--precision fp16:→ correct fp16 (
model.onnx.data≈ 242.6 MB, half of fp32's 485.2 MB).Input constraints (for perf / numeric — documented)
Random inputs crash this model; valid inputs are required:
token_type_idsmust be 0 (this RoBERTa-vocab checkpoint hastype_vocab_size = 1; index 1 is out of bounds in the token-type embedding).bboxmust be geometrically valid (x1 ≥ x0, y1 ≥ y0) so LayoutLM's height/width position embeddings receive non-negative indices. PyTorchnn.Embeddingraises on negative indices (ORT Gather silently wraps).Goal ladder (CPU, this run)
--precision fp16)input_ids/attention_mask/token_type_ids[1,512]+bbox[1,512,4]int32, outputsstart_logits[1,512]/end_logits[1,512]AutoModelForDocumentQuestionAnswering--ep allNumeric parity is clean. The new OnnxConfig correctly exports the extractive-QA head and the recipe fixes task/model-class detection.