fix: partition_email raises KeyError on a multipart/* attachment - #4424
fix: partition_email raises KeyError on a multipart/* attachment#4424mittalpk wants to merge 2 commits into
Conversation
There was a problem hiding this comment.
No issues found across 4 files
Shadow auto-approve: would auto-approve. Fixes a KeyError crash when partition_email encounters a multipart/* attachment by falling back to raw serialized bytes; includes a regression test.
Re-trigger cubic
cragwolfe
left a comment
There was a problem hiding this comment.
Draft review: Unstructured-IO/unstructured PR #4424
Verdict: Request changes
[High] Serialize multipart attachments as standalone documents
unstructured/partition/email.py:443
self._attachment.as_bytes() preserves the part's top-level Content-Disposition: attachment header. When an attachment is named *.eml, auto.partition() routes those serialized bytes back through partition_email(). The serialized part is now the recursive root message, and EmailMessage.get_body() excludes roots whose disposition is attachment, so a valid multipart attachment's body is silently dropped.
I reproduced this with a boundary-bearing multipart/signed attachment: the outer body was returned, but the signed message body was absent. This is introduced by the new serialization path and affects the PGP/MIME forwarded-message case described by the PR.
The added fixture does not exercise that behavior. Its multipart/signed part has no boundary and applies base64 transfer encoding to the multipart entity, so Python parses it as a scalar payload (is_multipart() == False, zero child parts). It proves that the original KeyError no longer escapes, but not that a well-formed multipart attachment is processed correctly.
Please serialize a copy of the multipart part as a standalone document by removing only the copy's top-level Content-Disposition before calling as_bytes(). Preserve the original part for filename metadata and do not mutate the parsed message tree. Add a boundary-bearing multipart/signed regression case that asserts:
- the outer email body is emitted;
- the signed attachment body is emitted;
- the signed body retains
filename == "signed-message.eml"and the enclosingattached_to_filenamemetadata.
The existing malformed fixture can remain as a separate no-crash regression.
(authored by codex)
eeshsaxena
left a comment
There was a problem hiding this comment.
Reasonable fix. Since email.contentmanager genuinely has no get_content() handler registered for any multipart/* type, catching that specific case in _AttachmentPartitioner._file_bytes and falling back to the part's raw serialized bytes is the right call, and it lets partition_email() recover the main body instead of the whole call dying on one signed/forwarded sub-part. The regression .eml with a multipart/signed attachment reproduces the original KeyError path, so the test genuinely guards it. Reads correct.
|
Good catch. Fixed in 1d1ca7e — the fix now strips |
email.contentmanager has no get_content() handler registered for any
multipart/* content-type (handlers exist for text/*, application/*,
image/*, message/rfc822, etc., but multipart sub-parts are normally
consumed via .iter_parts(), not .get_content()). A multipart sub-part
can still surface as an "attachment" via iter_attachments() though --
e.g. a PGP/MIME-signed forwarded message (multipart/signed) nested
inside a multipart/mixed envelope, or a multipart/mixed forwarded
message itself. Resolving such an attachment's bytes in
_AttachmentPartitioner._file_bytes raised KeyError('multipart/signed')
(or 'multipart/mixed'), crashing the entire partition_email() call
instead of processing or skipping just that one attachment.
Fixed by falling back to the MIME part's raw serialized bytes
(as_bytes()) when its content-type is multipart/*, rather than calling
get_content(). This is a root-cause fix at the actual failure point,
not a broadening of the shared, deliberately narrow
EXPECTED_ATTACHMENT_ERRORS tuple (its own comment: "Intentionally
narrow... we do not catch RuntimeError... would otherwise be silently
skipped") -- a bare KeyError there would mask unrelated bugs elsewhere
in partition().
Fixes Unstructured-IO#3922.
Serializing a multipart/* attachment with as_bytes() preserved its own Content-Disposition: attachment header (that's how it surfaced as an attachment via iter_attachments() in the first place). Since EmailMessage.get_body() skips any candidate part -- including a multipart/* root message -- whose own disposition is "attachment", re-parsing that serialized part as the root of a new message found no body at all: the attachment partitioned to zero elements instead of surfacing its own content (e.g. a PGP-signed forwarded message's body). _serialized_multipart_bytes now serializes a deep copy of the part with that header stripped, leaving the original attachment object (still needed for its filename) untouched. Also fixed the existing regression fixture (mime-attach-multipart-signed.eml): its multipart/signed part had Content-Transfer-Encoding: base64, which is invalid for a multipart entity per RFC 2045 -- Python's parser treated it as an opaque scalar payload with zero child parts, so the fixture could never have exercised this bug regardless of the fix. Rewrote it as a genuine boundary-bearing multipart/signed part and extended the test to assert the attachment's own body and metadata now come through, not just that no exception is raised.
1d1ca7e to
ba4935a
Compare
Fixes #3922
What's wrong
partition_email(..., process_attachments=True)crashes with an uncaughtKeyErroron emails containing a PGP-signed (or any othermultipart/*-typed) attachment — matching the traceback in #3922 exactly.Root cause: Python's
email.contentmanagerhas noget_content()handler registered for anymultipart/*content-type (handlers exist fortext/*,application/*,image/*,message/rfc822, etc., but multipart sub-parts are normally consumed via.iter_parts(), not.get_content()). A multipart sub-part can still surface as an "attachment" viaEmailMessage.iter_attachments()though — e.g. a PGP/MIME-signed forwarded message (multipart/signed) nested inside amultipart/mixedenvelope (the linkedtestcase.txtscenario), or amultipart/mixedpart itself (the exactKeyError: 'multipart/mixed'in the original traceback)._AttachmentPartitioner._file_bytes(unstructured/partition/email.py) unconditionally callsself._attachment.get_content(), which raisesKeyError(content_type)for these cases — and since that call happens before thetry/exceptin_iter_elements()that's meant to gracefully skip unpartitionable attachments, the exception propagates out of the wholepartition_email()call instead of being caught.Fix
_file_bytesnow checksself._attachment.get_content_type().startswith("multipart/")and falls back to the part's raw serialized bytes (.as_bytes()) in that case, instead of callingget_content(). This is a root-cause fix at the actual failure point.I deliberately did not widen
EXPECTED_ATTACHMENT_ERRORS(inunstructured/partition/common/__init__.py) to includeKeyErroras an alternative fix — that tuple's own comment says it's "Intentionally narrow" specifically because a bareKeyErrorcatch there would also silently swallow unrelated bugs elsewhere inpartition(). Fixing_file_bytesdirectly avoids that trade-off entirely.How was this tested?
.eml(amultipart/mixedemail with a nested PGP/MIME-signed forwarded message as an attachment, mirroring the issue's real-world scenario) before writing any fix — confirmed the exactKeyError('multipart/signed').test_partition_email_does_not_raise_on_multipart_attachment, using a new fixtureexample-docs/eml/mime-attach-multipart-signed.eml, following the existing pattern oftest_partition_email_silently_skips_attachments_it_cannot_partition(the MP3-attachment test). Confirmed the new test fails against unpatchedemail.py(git stash) with the exact reportedKeyError, and passes after the fix — the email body is still partitioned, and the unpartitionable attachment produces no elements without crashing.test_unstructured/partition/test_email.pysuite: 72 passed, no regressions.ruff check/ruff format --check— clean.scripts/version-sync.sh -c— clean (no version bump needed;CHANGELOG.mdupdated under the existing unreleased0.25.2-dev0section per the contributing checklist).Context on prior attempt
A previous PR (#4277) attempted a fix for this same issue but was self-closed by its author after 3 days with no maintainer engagement, and its fixtures never landed — confirmed the bug is still fully live on current
mainbefore starting this PR. This PR takes a narrower approach than that one (a single-point fix in_file_bytes, no changes to the sharedEXPECTED_ATTACHMENT_ERRORStuple or to_iter_elements's control flow).