fix: round-trip serialization of mixed-type lists in schema-based serialization

[davidpavlovschi]

Related Issues

• partially addresses Follow-Up: Schema-Based Serialization Improvements #9539 (the second follow-up there — promoting the schema serialization helpers to public utilities — is intentionally not part of this PR and the issue should stay open)

Proposed Changes:

_serialize_value_with_schema derived the array schema from the first element only, so any list/tuple/set holding mixed types was described by a schema that lied about every element after the first. Two concrete failures on main (commit 5add74e):

_deserialize_value_with_schema(_serialize_value_with_schema({"x": [Document(content="a"), "plain string", 3]}))
# AttributeError: 'str' object has no attribute 'copy'

_serialize_value_with_schema({"y": [1, "a", {"k": 2}]})["serialization_schema"]
# {'type': 'object', 'properties': {'y': {'type': 'array', 'items': {'type': 'integer'}}}}
# -> every element typed as integer, payload silently mis-typed

This hits real runtime values: Agent State fields and pipeline-breakpoint inputs holding heterogeneous lists crash or corrupt on resume.

Changes in haystack/utils/base_serialization.py:

• Serializer: compute each element's schema during the single existing pass instead of re-serializing the first element. When all element schemas are equal, emit the current {"type": "array", "items": {...}} envelope unchanged. Only when they differ, emit the JSON-Schema-standard {"type": "array", "prefixItems": [schema_per_item, ...]}. The uniqueItems / frozen / minItems / maxItems markers are attached exactly as before, so set/frozenset/tuple reconstruction is untouched.
• Deserializer: in the array branch, prefer prefixItems when present (each item is zipped with its own schema) and otherwise fall back to schema["items"] exactly as today. A prefixItems length that disagrees with the data length raises a DeserializationError instead of a raw zip error.
• Updated the two docstring notes that stated mixed-type lists are unsupported.
• Added a release note under releasenotes/notes/.

Backward compatibility: homogeneous arrays serialize to the byte-for-byte same dict as before, and payloads written by older versions (which only ever carry items) still deserialize through the unchanged items path. Both are asserted by dedicated tests.

How did you test it?

Unit tests added in test/utils/test_base_serialization.py (new TestMixedTypeArrays class): the exact serialized schema for mixed primitives, round-trips for mixed primitives, primitives mixed with a to_dict object (Document), a mixed tuple, mixed sets and frozensets (with container-type assertions), nested mixed lists, the empty list, plus a backward-compat assertion that a homogeneous list/tuple still serializes to the pre-change dict, that an old-format items-only payload still deserializes, and that a prefixItems/data length mismatch raises DeserializationError.

Verified red-before / green-after: with the new tests against the unmodified base_serialization.py, 5 of them fail (including the AttributeError crash); with the fix all pass.

Ran locally (Python 3.12, editable install):

• pytest test/utils/test_base_serialization.py test/core/pipeline/test_breakpoint.py test/core/pipeline/test_component_checks.py — the three call sites of these helpers: 160 passed.
• pytest test/utils test/core test/dataclasses -m "not integration" — 2534 passed, 4 failures all pre-existing and unrelated to this change (missing optional torch/ipython in this environment).
• ruff check and ruff format --check on both touched files — clean.
• mypy on haystack/utils/base_serialization.py — clean.

Notes for the reviewer

The issue offers "implement or document" as alternatives. This PR takes the implement route because the current behaviour is a hard crash (AttributeError leaking out of deserialization), not just a documented gap. The mechanism is deliberately conservative: nothing about homogeneous output changes, so existing snapshots and stored State payloads keep loading unchanged, and prefixItems is the standard JSON Schema keyword for positional element schemas rather than a Haystack-specific marker.

One behavioural detail worth a look: a set or frozenset with mixed types now round-trips too. The per-position schemas are produced from the same single iteration that produces the data, so the pairing is always consistent even though set iteration order is arbitrary.

This PR was fully generated with an AI assistant. I have reviewed the changes and run the relevant tests.

Checklist

• I have read the contributors guidelines and the code of conduct.
• I have added unit tests and updated the docstrings.
• I've used one of the conventional commit types for my PR title: fix:, feat:, build:, chore:, ci:, docs:, style:, refactor:, perf:, test: and added ! in case the PR includes breaking changes.
• I have documented my code.
• I have added a release note file, following the contributors guidelines.
• I have run pre-commit hooks and fixed any issue.

[vercel]

@davidpavlovschi is attempting to deploy a commit to the deepset Team on Vercel.

A member of the Team first needs to authorize it.

[CLAassistant]

All committers have signed the CLA.

[sjrl]

Hey @davidsbatista I can take review of this one. I was recently working on this.