feat: add Runner, RunnerResult, Judge, and Evaluator

[mattrmc1]

Summary

Adds the AIEVALS types — Runner, RunnerResult, Judge, and Evaluator — and wires Evaluator.noop() into all config types. Callers can now implement a Runner to wrap any model provider, construct a Judge to evaluate AI outputs against a judge prompt with structured {score, reasoning} output, and coordinate multiple judges through an Evaluator.

New types

public interface Runner {
  RunnerResult run(String input, Map<String, Object> outputType) throws Exception;
  default RunnerResult run(String input) throws Exception;
}

Wraps a model provider SDK. outputType carries a JSON-Schema-like map when structured output is needed. Single-arg overload delegates with outputType = null.

RunnerResult.builder(String content, AIMetrics metrics)
    .raw(Object raw)
    .parsed(Map<String, Object> parsed)
    .build();

Immutable result of a Runner invocation. parsed is defensively copied and returned as unmodifiable.

public Judge(AIJudgeConfig config, Runner runner, LDLogger logger);

JudgeResult evaluate(String input, String output);
JudgeResult evaluate(String input, String output, double samplingRate);
JudgeResult evaluateMessages(List<Message> messages, RunnerResult response);
JudgeResult evaluateMessages(List<Message> messages, RunnerResult response, double samplingRate);

Evaluates AI output by invoking a runner with a formatted evaluation prompt and parsing the structured response. Sampling gate runs first — below the rate, returns sampled=false immediately. Creates a fresh tracker per evaluation via config.createTracker(). Parses score (Number, [0.0, 1.0]) and reasoning (String, optional). Runner exceptions are caught and returned as JudgeResult(success=false) — judge failures are results, not exceptions. Does not call trackJudgeResult.

public static Evaluator noop();
public Evaluator(Map<String, Judge> judges, JudgeConfiguration judgeConfiguration, LDLogger logger);

CompletableFuture<List<JudgeResult>> evaluate(String input, String output);

Coordinates sequential execution of judges. Missing judges skipped with a warning. Evaluator.noop() returns a singleton whose evaluate immediately returns an empty list. For v1.0, all configs receive Evaluator.noop().

Config type changes

AIConfig base class gains an Evaluator field and getEvaluator() accessor. AICompletionConfig and AIAgentConfig constructors accept an Evaluator. AIJudgeConfig always wires Evaluator.noop() internally — judges do not evaluate themselves.

Test plan

• ./gradlew :lib:sdk:server-ai:test passes
• JudgeTest — successful evaluation, score boundary validation (0 and 1), reasoning optional, runner exception handling (caught not rethrown), null/missing parsed output, score out of range, sampling rates (0 always skips, 1 always runs), message formatting, getter accessors
• EvaluatorTest — noop returns empty list, noop singleton identity, single/multiple judge execution, missing judge skipped, evaluator does not call trackJudgeResult, returned future is already complete
• RunnerResultTest — builder field assignment, immutability, defensive copy of parsed map

---

Note

Medium Risk
New public SDK surface and constructor wiring on config types; runtime behavior stays noop until real evaluators are wired, but Judge paths will invoke Runners and extra model calls when enabled later.

Overview
Introduces the AI evaluation stack for the server AI SDK: a Runner abstraction (plus immutable RunnerResult) for provider calls, a Judge that runs judge prompts via a runner and parses structured {score, reasoning} with sampling and defensive failure handling, and an Evaluator that runs configured judges sequentially and returns CompletableFuture<List<JudgeResult>> (missing judges are skipped with a warning; trackJudgeResult is left to callers).

AIConfig now carries an Evaluator exposed via getEvaluator(). AICompletionConfig and AIAgentConfig take an evaluator in their package-private constructors; LDAIClientImpl always supplies Evaluator.noop() for retrieved configs in this release. AIJudgeConfig passes Evaluator.noop() through the base class because judges do not self-evaluate.

Unit tests cover Judge, Evaluator, and RunnerResult behavior (sampling, parsing, immutability, noop paths).

^{Reviewed by Cursor Bugbot for commit 25346b3. Bugbot is set up for automated code reviews on this repo. Configure here.}

[cursor]

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit faa4981. Configure here.}