Improve `null` handling

[dittos]

Note

Handling "absent" fields are out of scope of this issue as it's not supported by OpenAI structured outputs. https://platform.openai.com/docs/guides/structured-outputs#all-fields-must-be-required

Things to consider

• How nullable values are represented as type hints
• Which callbacks are called in which order, on null values
• Derivation of Pydantic type hint

Notation:

• T: str, int, float, bool, or Pydantic model class
• S: StreamingValue subclass

non-streaming nullable values

ld.Atom[T | None]

nullable streaming objects

ld.Optional[S]

nullable streaming strings

ld.Optional[ld.String]

nullable streaming lists

• ld.Optional[ld.List[T]]
• ld.Optional[ld.List[S]]

streaming lists with nullable non-streaming items

ld.List[T | None]

streaming lists with nullable streaming items

ld.List[ld.Optional[S]]

ld.Optional[S] API design

class Item(ld.Object):
  field: ld.Optional[ld.String]

item = Item()


# Non-null callbacks are available under `.value` (is `ld.String` here)
#  alternative namings: .some, .non_null, .present, .exist, .available, ...

@item.field.value.on_append
def on_chunk(chunk: str):
  pass

@item.field.value.on_complete
def on_complete(value: str):
  pass


# Handle null in `on_complete` callback

@item.field.on_complete
def on_complete(value: ld.String | None): 
  # note: value is not `str | None` because extracting `str` from `ld.String` is not possible in Python
  pass

# or separate callback for null?

@item.field.on_null
def on_null():
  pass