fix: use flatten=False in Document.__eq__ to avoid metadata key collisions - #12172
fix: use flatten=False in Document.__eq__ to avoid metadata key collisions#12172rekha0suthar wants to merge 1 commit into
Conversation
…sions Document.__eq__ called to_dict() with the default flatten=True, which merges metadata into the top-level dictionary. When a Document's metadata contains a key that collides with a real Document field (e.g. 'id' or 'score'), the meta value overwrites the field value before comparison, causing two Documents with different metadata to compare as equal. Switching to to_dict(flatten=False) compares the structured representation directly, so meta is kept separate from top-level fields and collisions can no longer produce false positives. Added a regression test covering the case where two Documents share the same content but differ only in a meta key named 'id'. Fixes deepset-ai#11969
|
@rekha-suthar is attempting to deploy a commit to the deepset Team on Vercel. A member of the Team first needs to authorize it. |
|
|
|
Hi @rekha0suthar, thanks for your interest in contributing to Haystack! 🙏 This is an automated message to help us keep the review queue healthy. |
|
Closing as a duplicate of #11982 which is already addressing this issue. Thanks for the heads up! |
Summary
Document.__eq__calledto_dict()with the defaultflatten=True, which merges the document'smetadictionary into the top-level representation. When a document's metadata contains a key that shares a name with a real Document field (such as"id"or"score"), the meta value silently overwrites the field value before the comparison is made. This causes two Documents with different metadata to compare as equal.Example that was broken before this fix:
Fix
Changed line 97 of
haystack/dataclasses/document.pyfrom:to:
flatten=Falsekeeps themetadictionary nested under its own key, so collisions between meta keys and top-level field names cannot affect the comparison result.Tests
Added
test_equality_with_colliding_meta_keystotest/dataclasses/test_document.pyto prevent regressions on this exact case.Fixes #11969