Skip to content

fix(cleaners): handle empty text in ordered-bullet cleaners - #4425

Open
asjad3 wants to merge 2 commits into
Unstructured-IO:mainfrom
asjad3:fix/ordered-bullets-empty-text
Open

fix(cleaners): handle empty text in ordered-bullet cleaners#4425
asjad3 wants to merge 2 commits into
Unstructured-IO:mainfrom
asjad3:fix/ordered-bullets-empty-text

Conversation

@asjad3

@asjad3 asjad3 commented Aug 2, 2026

Copy link
Copy Markdown

Summary

clean_ordered_bullets() and extract_ordered_bullets() read the first token of
text.split() without checking that a token exists, so empty or whitespace-only text
raises IndexError instead of passing through:

>>> from unstructured.cleaners.core import clean_ordered_bullets
>>> clean_ordered_bullets("")
IndexError: list index out of range

Every other cleaner in unstructured/cleaners/core.py returns '' for empty input —
clean_bullets, clean_extra_whitespace, clean_dashes, clean_trailing_punctuation,
clean_non_ascii_chars, clean_ligatures, replace_unicode_quotes,
remove_punctuation — so this looks like an oversight rather than an intended contract.

It matters because elements with empty text are common, and cleaners are applied per
element. A single empty element aborts the whole run:

from unstructured.documents.elements import Text
from unstructured.cleaners.core import clean_ordered_bullets

for el in [Text("1.1 Introduction"), Text(""), Text("2.1 Methods")]:
    el.apply(clean_ordered_bullets)   # raises IndexError on the empty one

Changes

Both functions now return early when text.split() is empty — the text unchanged for
clean_ordered_bullets, and (None, None, None) for extract_ordered_bullets, which is
what it already returns for any text that carries no bullet.

Testing

The snippet above runs clean on this branch. Or via the existing parametrized tests,
which now cover "", " " and "\n" for both functions:

pytest test_unstructured/cleaners/test_core.py test_unstructured/cleaners/test_extract.py
  • 133 passed on this branch (127 before, plus the 6 new cases)
  • The 6 new cases fail on main with IndexError, pass here
  • ruff check / ruff format --check clean on the changed files

CHANGELOG and __version__.py bumped to 0.25.2-dev1.


Written with AI assistance (Claude Code); I reviewed the change and ran the checks above.

Review in cubic

clean_ordered_bullets() and extract_ordered_bullets() read text.split()[0]
without checking that a token existed, so empty or whitespace-only text
raised IndexError instead of being passed through.

Every other cleaner in the module returns '' for empty input, and elements
with empty text are common, so applying either cleaner across a document
(element.apply(clean_ordered_bullets)) aborted the whole run on the first
empty element.

Return the text unchanged and (None, None, None) respectively.

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No issues found across 6 files

Shadow auto-approve: would auto-approve. Focused bug fix: ordered-bullet cleaners now return early on empty/whitespace text instead of raising IndexError, matching sibling cleaners. New parameterized tests pin the corrected behavior; change is bounded and clearly beneficial.

Re-trigger cubic

@eeshsaxena eeshsaxena left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Clean fix, and it looks complete for this pattern rather than a spot patch. The unguarded text.split()[0] / [1:] indexing only lives in clean_ordered_bullets (core.py) and extract_ordered_bullets (extract.py), and both are guarded here, so there is no sibling in the cleaners module still exposed to the same IndexError. Returning the text unchanged / (None, None, None) on empty input also matches how the other cleaners in the module behave on no-op input, so element.apply(clean_ordered_bullets) over a document with an empty element no longer aborts the whole run. The added empty/whitespace/newline test cases cover it well.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants