Skip to content

fix: better handling of params and custom params for optimization#163

Open
andrewklatzke wants to merge 1 commit intoaklatzke/AIC-2263/sdk-dx-improvementsfrom
aklatzke/AIC-2426/optimization-params-feedback
Open

fix: better handling of params and custom params for optimization#163
andrewklatzke wants to merge 1 commit intoaklatzke/AIC-2263/sdk-dx-improvementsfrom
aklatzke/AIC-2426/optimization-params-feedback

Conversation

@andrewklatzke
Copy link
Copy Markdown
Contributor

@andrewklatzke andrewklatzke commented May 4, 2026

Requirements

  • I have added test coverage for new or changed functionality
  • I have followed the repository's pull request submission guidelines
  • I have validated my changes against all supported platform versions

Describe the solution you've provided

Implements better handling for params on the initial variation (folds model changes in as overwrites rather than completely replacing) and ensures custom params are persisted unchanged. Additionally makes sure that tools cannot be changed by the optimization process.

Describe alternatives you've considered

This is the result of a bug report so there weren't really alternatives considered.

Additional context

Initially it was assumed that the optimization process would properly pull params forward (via the LLM) but this doesn't seem to always be the case. In the case of custom params, they aren't fed into the LLM calls since they're user-specified data (not specific to the actual optimization result). We now just pull these through as-is. In the case of tools, the model will be able to optimize the prompt to call a specific tool if multiple are provided, but we don't want to strip any tool information from the final result as it may be necessary for the calls to function.


Note

Medium Risk
Moderate risk because it changes how model parameters and tools are carried forward across optimization iterations and what gets auto-committed, which can affect runtime agent behavior if merging/restoration logic is wrong.

Overview
Improves variation-application logic so LLM-generated current_parameters are merged into existing parameters instead of replacing them, preserving user-specified/custom settings (e.g. max_tokens, response_format) when the LLM omits them.

Prevents tool drift by always restoring the original tools list (and logging when the LLM returns a different one) to avoid silently dropping user tools or leaking internal framework tools.

Captures model.custom from the initial LaunchDarkly variation and includes it when auto-committing a winning variation; adds focused test coverage for parameter persistence, tool restoration/warnings, and model.custom propagation.

Reviewed by Cursor Bugbot for commit 572a2aa. Bugbot is set up for automated code reviews on this repo. Configure here.

@andrewklatzke andrewklatzke requested a review from jsonbailey May 4, 2026 23:47
@andrewklatzke andrewklatzke requested a review from a team as a code owner May 4, 2026 23:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants