Added support to handle nested Pydantic types

[AdilFayyaz]

Tracking issue

Duplicated from: flyteorg/flyte-sdk#640

Why are the changes needed?

To ensure the same behavior persists between v1 and v2

What changes were proposed in this pull request?

• Fix _get_element_type() in the type engine to correctly resolve nested and recursive Pydantic types that were previously flattened to str

• Add $ref resolution so nested models or enums inside containers (List, Dict) are resolved via $defs lookup instead of defaulting to str

• Rewrite anyOf handling to support $ref variants (e.g. Optional[Inner]), fixing KeyError: 'type' crashes

• Add recursive array/object branches so nested containers (List[List[int]], Dict[str, Dict[str, int]]) resolve to their correct Python types

• Handle null JSON schema type, returning NoneType instead of str

• Add example demonstrating all nested Pydantic type patterns running as parallel Flyte tasks

• Note: In v1, _handle_json_schema_property is called recursively for each variant inside an anyOf (e.g. Optional[Inner] produces {"anyOf": [{"$ref": "#/$defs/Inner"}, {"type": "null"}]}). When it recurses on the $ref variant {"$ref": "#/$defs/Inner"}, there is no "type" key, causing a KeyError: 'type'.

• v1 iterates over all properties, so we added $ref resolution in _handle_json_schema_property before the "type" lookup to delegate to _get_element_type, which already knows how to resolve $ref paths.

How was this patch tested?

• pytest tests/flytekit/unit/core/test_nested_structs_in_pydantic.py: 16 new tests covering -
• Nested arrays: List[List[int]] schema resolution + roundtrip, List[List[Inner]] roundtrip
• Nested dicts: List[Dict[str, int]] schema + roundtrip, Dict[str, Dict[str, int]] schema + roundtrip, List[Dict[str, Inner]] roundtrip, Dict[str, Dict[str, Inner]] roundtrip
• anyOf with $ref: List[Optional[Inner]] schema resolution (no KeyError), List[Optional[str]] schema resolution, List[Optional[Inner]] roundtrip via guess_python_type
• null type: {"type": "null"} resolves to NoneType
• Enums in containers: List[ModelWithEnum] roundtrip, Dict[str, ModelWithEnum] roundtrip
• Complex combined: model with List[List[Inner]], Dict[str, Inner], List[Dict[str, int]], List[ModelWithEnum], Optional[Inner] roundtrip with data and with None optional

Setup process

Screenshots

Check all the applicable boxes

• I updated the documentation accordingly.
• All new and existing tests passed.
• All commits are signed-off.

Related PRs

Docs link

[welcome]

Thank you for opening this pull request! 🙌

These tips will help get your PR across the finish line:

• Most of the repos have a PR template; if not, fill it out to the best of your knowledge.
• Sign off your commits (Reference: DCO Guide).