Skip to content

chore: skip tracking judge results that were not sampled#167

Merged
jsonbailey merged 1 commit intomainfrom
jb/skip-unsampled-judge-results
May 5, 2026
Merged

chore: skip tracking judge results that were not sampled#167
jsonbailey merged 1 commit intomainfrom
jb/skip-unsampled-judge-results

Conversation

@jsonbailey
Copy link
Copy Markdown
Contributor

@jsonbailey jsonbailey commented May 5, 2026

Summary

  • Skip processing judge results where r.sampled is False in ManagedModel._run_and_track
  • Prevents unsampled evaluations from being tracked unnecessarily

Test plan

  • Verify judge results with sampled=False are skipped without errors
  • Verify judge results with sampled=True continue to be tracked as before

🤖 Generated with Claude Code


Note

Low Risk
Low risk: adds a simple sampled gate before tracking judge results, affecting only metrics/logging for evaluations that were intentionally skipped by sampling.

Overview
Judge result tracking now ignores unsampled evaluations. In ManagedModel and ManagedAgent, _run_and_track skips any JudgeResult where sampled is False, so only actually-executed judge runs are passed to tracker.track_judge_result (and related warnings).

Reviewed by Cursor Bugbot for commit e45e7f7. Bugbot is set up for automated code reviews on this repo. Configure here.

@jsonbailey jsonbailey marked this pull request as ready for review May 5, 2026 23:08
@jsonbailey jsonbailey requested a review from a team as a code owner May 5, 2026 23:08
Copy link
Copy Markdown

@cursor cursor Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 34624ba. Configure here.

Comment thread packages/sdk/server-ai/src/ldai/managed_model.py
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@jsonbailey jsonbailey force-pushed the jb/skip-unsampled-judge-results branch from 34624ba to e45e7f7 Compare May 5, 2026 23:36
@jsonbailey jsonbailey merged commit 3996073 into main May 5, 2026
45 checks passed
@jsonbailey jsonbailey deleted the jb/skip-unsampled-judge-results branch May 5, 2026 23:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants