Check signature of returned Callable for ProgramSynthesisHander

[datvo06]

Closes #361

[jfeser]

This doesn't really address #361, which is about checking the type-correctness of the function. A solution to #361 needs to run some type checker (e.g. mypy).

I think a type checker based version would be both simpler and more robust.

[datvo06]

@jfeser Thanks! I see, so we directly do the static analysis. Is the expected solution somewhere along this line?

import tempfile
from mypy import api

def _type_check_code(self, code: str, expected_type: type) -> None:
    """Run mypy on generated code to verify type correctness."""
    # Write code to temp file for mypy
    with tempfile.NamedTemporaryFile(mode='w', suffix='.py', delete=False) as f:
        f.write(code)
        temp_path = f.name
    
    try:
        result = api.run([temp_path, '--no-error-summary', '--no-pretty'])
        stdout, stderr, exit_code = result
        
        if exit_code != 0:
            raise SynthesisError(f"Type check failed:\n{stdout}", code)
    finally:
        import os
        os.unlink(temp_path)

[jfeser]

I suspect the hard part of #361 will be giving the type checker appropriate context so that it can check code that uses types not defined by the LLM. E.g.

class User:
    name: str
    age: int
    salary: float

@Template.define
def _validation() -> Callable[[User], bool]:
    '''Write a function to check that users named Jack make at least $200000'''
    ...

The type checker needs to see the definition of User.

[datvo06]

@jfeser I see, thanks! I think I got a better idea now. Let me try another iteration.
Additionally, I think the synthesis handler would also need to quote the type, maybe recursively to the context of generation?

[datvo06]

@jfeser I made another attempt in #422

[eb8680]

I think this may have been handled in #519