fix: nested DOCX sub-lists are being converted to un-nested lists in MarkDown - #2324
Open
org30h wants to merge 1 commit into
Open
fix: nested DOCX sub-lists are being converted to un-nested lists in MarkDown#2324org30h wants to merge 1 commit into
org30h wants to merge 1 commit into
Conversation
…inition Word can express a nested list either as a deeper w:ilvl within the parent's w:numId, or as a new w:numId at w:ilvl 0 that is set apart only by its indentation. Both render identically in Word, but mammoth derives nesting from w:ilvl alone, so the second form was flattened into the parent list and its items were renumbered as siblings. Extend the existing pre_process_docx step to resolve each level's effective indentation from numbering.xml and walk the document tracking the open list levels, so nesting implied by indentation is restored before mammoth reads the file. Within one w:numId the declared w:ilvl stays authoritative, since some numbering definitions give several levels the same indentation. Indentation only ever adds nesting that the declared levels missed and never removes nesting a document states outright, which leaves documents that already convert correctly untouched. Remapped paragraphs are pointed at a generated w:abstractNum carrying their original w:numFmt, so a bulleted sub-list is not silently converted into a numbered one, and depth is capped at the last level mammoth's default style map defines. Fixes microsoft#2323
Author
|
@microsoft-github-policy-service agree |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #2323
Root cause
MarkItDown converts DOCX through mammoth, and mammoth decides list nesting from the
w:ilvlvalue on each paragraph. Word does not always usew:ilvlto express nesting. Indenting a list in Word will sometimes start a brand new numbering definition instead, giving the sub-list its ownw:numIdatw:ilvl0 and setting it apart from its parent only by indentation.Both forms look identical in Word. mammoth understands only the first one. In the second form the sub-list carries the same
w:ilvlas its parent, so mammoth reads the items as siblings, flattens them into the parent list and renumbers them.In the document attached to #2323, items 3.1 and 3.2 use
numId3 atilvl0 while item 3 usesnumId1 atilvl0. They came out as top level items 4 and 5.What this changes
Everything happens inside
pre_process_docx, which already rewritesword/document.xmlbefore mammoth reads it. The converter is untouched and so is mammoth. The diff against the existing function is a 14 line insertion, and the rest of the change is new helper functions.The new step reads the effective indentation of every
(numId, ilvl)pair fromword/numbering.xml, then walks the document in order while tracking which list levels are currently open. A paragraph indented further than the level above it is a nested item regardless of what its declaredw:ilvlsays. Body text closes any open levels, which matches how mammoth already ends a list.Three details carry most of the correctness.
Inside a single
numIdthe declaredw:ilvlstays authoritative and indentation is consulted only across differentnumIdvalues. This matters because some numbering definitions give two levels the same indentation, and comparing indentation alone would flatten lists that are already correct today.Indentation is only ever used to add nesting that the declared levels missed. It never removes nesting that a document states outright. That property is what keeps every document mammoth already handles from changing at all.
Remapped paragraphs are not simply given a higher
w:ilvl. Doing that would make the paragraph resolve against whatever unrelated level its numbering happens to define at that index, so a bulleted sub-list whose numbering defines level 1 asdecimalwould quietly become a numbered list. Instead each remapped combination gets a small generatedw:abstractNumthat carries the paragraph's originalw:numFmtto the new depth.Depth is capped at the fifth level because mammoth's default style map stops there. A paragraph promoted past it matches no rule and falls out of the list entirely.
The whole step is wrapped in
try/except. If anything goes wrong the original XML is used and the output is exactly what it is today.Tests
Added
packages/markitdown/tests/test_docx_lists.pywith seven tests, and the file from #2323 as themultilevel_lists.docxfixture. Tests build small DOCX files in memory and assert on the output of the publicMarkItDownAPI, so they do not depend on how the fix is implemented.Three of them fail on
mainand pass with this change:test_docx_multilevel_liststest_docx_sub_list_as_new_num_idnumIdatilvl0test_docx_sub_list_preserves_bulletsilvl1 asdecimalFour pass both with and without the change. They are there to show the fix does not disturb lists that already convert correctly:
test_docx_nesting_from_declared_levels_is_unchangedw:ilvlnesting still converts the same waytest_docx_equal_level_indents_are_not_flattenedtest_docx_list_interrupted_by_paragraphtest_docx_deep_nesting_keeps_every_itemVerification
Run on Python 3.12 with
packages/markitdown[all]installed, which matches what CI uses.The full suite was run twice, once on
mainand once with this branch, and the set of failing tests is byte for byte identical. Every failure in that set is environmental and pre-existing, from Windows path handling intest_file_urisand from the CLI tests resolving a subprocess interpreter that has nomarkitdowninstalled. None of them involve DOCX conversion.With remote tests skipped the way CI skips them, and excluding that CLI subprocess group, the result is 272 passed and 22 skipped, with the single Windows only path failure left.
file_uri_to_pathwas checked directly under Linux and returns the POSIX paths that test expects, so it passes on theubuntu-latestrunner.The seven new tests were run on Python 3.12 and 3.14 and pass on both.
pre-commit run --all-filespasses.