fix(cleaners): handle empty text in ordered-bullet cleaners - #4425
fix(cleaners): handle empty text in ordered-bullet cleaners#4425asjad3 wants to merge 2 commits into
Conversation
clean_ordered_bullets() and extract_ordered_bullets() read text.split()[0] without checking that a token existed, so empty or whitespace-only text raised IndexError instead of being passed through. Every other cleaner in the module returns '' for empty input, and elements with empty text are common, so applying either cleaner across a document (element.apply(clean_ordered_bullets)) aborted the whole run on the first empty element. Return the text unchanged and (None, None, None) respectively.
There was a problem hiding this comment.
No issues found across 6 files
Shadow auto-approve: would auto-approve. Focused bug fix: ordered-bullet cleaners now return early on empty/whitespace text instead of raising IndexError, matching sibling cleaners. New parameterized tests pin the corrected behavior; change is bounded and clearly beneficial.
Re-trigger cubic
eeshsaxena
left a comment
There was a problem hiding this comment.
Clean fix, and it looks complete for this pattern rather than a spot patch. The unguarded text.split()[0] / [1:] indexing only lives in clean_ordered_bullets (core.py) and extract_ordered_bullets (extract.py), and both are guarded here, so there is no sibling in the cleaners module still exposed to the same IndexError. Returning the text unchanged / (None, None, None) on empty input also matches how the other cleaners in the module behave on no-op input, so element.apply(clean_ordered_bullets) over a document with an empty element no longer aborts the whole run. The added empty/whitespace/newline test cases cover it well.
Summary
clean_ordered_bullets()andextract_ordered_bullets()read the first token oftext.split()without checking that a token exists, so empty or whitespace-only textraises
IndexErrorinstead of passing through:Every other cleaner in
unstructured/cleaners/core.pyreturns''for empty input —clean_bullets,clean_extra_whitespace,clean_dashes,clean_trailing_punctuation,clean_non_ascii_chars,clean_ligatures,replace_unicode_quotes,remove_punctuation— so this looks like an oversight rather than an intended contract.It matters because elements with empty text are common, and cleaners are applied per
element. A single empty element aborts the whole run:
Changes
Both functions now return early when
text.split()is empty — the text unchanged forclean_ordered_bullets, and(None, None, None)forextract_ordered_bullets, which iswhat it already returns for any text that carries no bullet.
Testing
The snippet above runs clean on this branch. Or via the existing parametrized tests,
which now cover
""," "and"\n"for both functions:mainwithIndexError, pass hereruff check/ruff format --checkclean on the changed filesCHANGELOG and
__version__.pybumped to0.25.2-dev1.Written with AI assistance (Claude Code); I reviewed the change and ran the checks above.