Skip to content

Fix Python remote parameter list serialization#206

Open
Joshua Wootonn (joshuawootonn) wants to merge 1 commit into
mainfrom
codex/fix-python-remote-params-list
Open

Fix Python remote parameter list serialization#206
Joshua Wootonn (joshuawootonn) wants to merge 1 commit into
mainfrom
codex/fix-python-remote-params-list

Conversation

@joshuawootonn
Copy link
Copy Markdown

@joshuawootonn Joshua Wootonn (joshuawootonn) commented May 27, 2026

Summary

I don't think the python remote eval case was tested in the bt implementation of parameters. The problem was that we were trying serialize the new internal type (RemoteEvalParameters) I created in v0.10.0 of the python parameters implementation with the old serializer (parameters_to_json_schema). I updated the code to use the new serializer and everything works now. It was already working for the vanilla python SDK.

Description

Customer logs showed /list endpoint for bt run remote evals was 500'ing when using saved Parameters.

[bt eval dev:list] stderr: AttributeError: 'RemoteEvalParameters' object has no attribute 'items'
...
File "/Users/divitoa/.cache/bt/eval-runners/0.8.0/eval-runner.py", line 570, in build_eval_definitions
  "parameters": parameters_to_json_schema(evaluator.parameters) if evaluator.parameters else {},
File "/Users/divitoa/code/pine/morgan/.venv/lib/python3.14/site-packages/braintrust/parameters.py", line 430, in parameters_to_json_schema
  for name, schema in parameters.items():
                    ^^^^^^^^^^^^^^^^

I created a reproduction (see demos) and found that bt run remote evals were also not working for static parameters. Instead of a 500 though, it was just showing a remote eval empty state in the playground.

After investigating, I realized that the DTO returned from the static parameter case was different for bt than it was for the vanilla Python SDK. That helped me realize that this implementation wasn't taking into account the new containerized DTO @ibolmo and I designed back when I was first working on the Parameter's API. Swapping parameters_to_json_schema to serialize_remote_eval_parameters_container was the solution, and fixed both cases by returning a DTO the playground understands from the /list endpoint.

Demo

Me going through the problem (sorry for the low quality on this video)

CleanShot.2026-05-27.at.14.18.15.mp4

The working thing:

CleanShot.2026-05-27.at.14.28.21.mp4

Old Python SDKs work in and outside of bt:

CleanShot.2026-05-27.at.14.58.09.mp4

@github-actions
Copy link
Copy Markdown

github-actions Bot commented May 27, 2026

Latest downloadable build artifacts for this PR commit 88e0f70f9092:

Available artifact names
  • ``artifacts-build-global
  • ``artifacts-build-local-x86_64-apple-darwin
  • ``artifacts-build-local-x86_64-pc-windows-msvc
  • ``artifacts-build-local-aarch64-pc-windows-msvc
  • ``artifacts-build-local-x86_64-unknown-linux-musl
  • ``artifacts-build-local-x86_64-unknown-linux-gnu
  • ``artifacts-build-local-aarch64-apple-darwin
  • ``artifacts-build-local-aarch64-unknown-linux-gnu
  • ``artifacts-plan-dist-manifest
  • ``cargo-dist-cache

@joshuawootonn Joshua Wootonn (joshuawootonn) force-pushed the codex/fix-python-remote-params-list branch 4 times, most recently from ef30419 to 3eaaac8 Compare May 27, 2026 16:37
@joshuawootonn Joshua Wootonn (joshuawootonn) force-pushed the codex/fix-python-remote-params-list branch from 3eaaac8 to 88e0f70 Compare May 27, 2026 19:06
Comment thread scripts/eval-runner.py
if not parameters:
return {}
if serialize_remote_eval_parameters_container is not None:
return serialize_remote_eval_parameters_container(parameters)
Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This function was shipped with the release of the Python parameters SDK, So its presence as an import will indicate that we have the right parameter type for serializing into a containerized remote eval value.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants