Add RFC 0007: Scorer Presets with Customization and Persistence

[Nehanth]

Summary

This RFC proposes a Preset class for MLflow that packages a named, immutable collection of scorers for common evaluation patterns. MLflow ships three built-in presets (Rag, Agent, ConversationalAgent) as starting points. Users can create custom presets, persist them to the MLflow server, and share them across teams and sessions.

from mlflow.genai.scorers import Agent, Guidelines

# Use a built-in preset
mlflow.genai.evaluate(
    data=eval_dataset,
    scorers=[Agent()],
)

# Create and persist a custom preset
my_preset = Preset("my_team_eval", scorers=[Safety(), Fluency(), my_custom_scorer])
my_preset.register()

# Load it later
preset = get_preset(name="my_team_eval")
mlflow.genai.evaluate(data=eval_dataset, scorers=[preset])

Based on mlflow/mlflow#21445.

Co-Authored-By: Claude noreply@anthropic.com

[mprahl]

Does it make sense to deduplicate scorers when combining presets especially since some have overlap? Edit: I see this is addressed below. It might be good to mention it here too.

[Nehanth]

Yes, I've added deduplication in add so duplicates are removed when presets are combined.

[mprahl]

We should keep Preset deduplicated whenever scorers are added instead of relying on validate_scorers to do it IMO.

[Nehanth]

I think we should have both, add (I added the deduplication there as well) handles deduplication when a user combines presets or scorers using the + operator, and validate_scorers() handles it when multiple presets are passed directly in the scorers list.

[mprahl]

This RFC has too many implementation details. The RFC should be scoped to requirements, APIs, database schema changes, etc. This is delving into PR level implementation.

Can we keep this scoped to the desired UX, the proposed groupings, and the class definition?

[Nehanth]

Trimmed it down!

[mprahl]

How does deduplication account for scorers with different configurations (e.g. ConversationalGuidelines with separate configs)?

[Nehanth]

Deduplication works by (type(scorer), scorer.name), so if you use different names for your
ConversationalGuidelines scorers, they'll be kept. If two scorers have the same type and name, one gets removed.

[mprahl]

Is this relevant for this RFC? It seems out of scope.

[Nehanth]

You're right, it's out of scope for this RFC. I'll remove it. I did create a separate issue for it mlflow#22972, once it's implemented it would be added to the Agent, ConversationalAgent, and Quality presets.comm

[mprahl]

I don't really see a benefit to these presets if you can't configure the scorers. If it's merely to categorize different scorers, there are better ways to do this such as documentation or a function like get_agent_scorers() which returns the classes for discoverability.

[Nehanth]

The reason I’m going in this direction is because, after this RFC is approved, subclassing opens up more than just categorization. For example, each preset could accept different judge models (e.g, Agent(model="openai:/gpt-4o")), different inference parameters, or other preset-specific configurations.

If this RFC gets approved, I can also update it to use the function-based approach I proposed in the alternatives section, if that’s preferred.

[B-Step62]

I agree with Matt about customization capability. The class-based approach is more ergonomic to me but I think it should come with a good customizability and persistence.

[mprahl]

Doesn't this require expected_response or expected_facts?

[Nehanth]

Yes, RetrievalSufficiency does require ground truth, so I'm removing it from this preset. We want presets to work out of the box without requiring ground truth for easy evaluation.

[mprahl]

Do you think it'd be less surprising to implement __or__ for "set" union behavior since you are deduplicating?

[Nehanth]

Yeah, Just did that!

[mprahl]

I’m not convinced this should be a feature in its current form. Without a way to configure or cleanly override scorers inside a preset, the value seems limited to boilerplate reduction. At the same time, if built-in preset membership changes across releases, users will get silent behavior and cost changes on upgrade. That combination makes presets feel more like unstable convenience sugar than a strong API surface.

[B-Step62]

Thank you for the RFC submission @Nehanth!

The idea of bundling multiple scorers into sharable preset sounds very useful to me. My main feedback is to support persistence of presets in MLflow server and provide great customization UX. I can see built-in presets are useful for new users to getting started and solves "which scorer should I start with?" question, but many teams need customization for adopting them for the real use cases.

[B-Step62]

I believe we need to persist presets more than a single python session. Otherwise L65 is essentially same as defining a normal python list of scorers. Team sharing benefit mentioned below only holds with persistence in a shared server.

[B-Step62]

nit: Instead of deduplication, I would block adding an existing scorer in the preset.

[B-Step62]

I agree with Matt about customization capability. The class-based approach is more ergonomic to me but I think it should come with a good customizability and persistence.

[B-Step62]

RAG/Agent/Conversation presets look great, the other two look less useful or intuitive to me. Safety one just bundles two scorers with 'safety' in its name. "Quality" sounds a bit too vague imo, technically most of scorers aims to evaluate some aspect of the agent quality. Shall we start with the first three + great customization story?

[etirelli]

I fully agree with @mprahl on this:

if built-in preset membership changes across releases, users will get silent behavior and cost changes on upgrade. That combination makes presets feel more like unstable convenience sugar than a strong API surface.

If this is not handled properly, this feature would end up being a burden for us to maintain, and create friction for users on upgrades, rather than being a benefit.
One way to solve this issue is to implement persistence as suggested by @B-Step62 , as that would empower users to control how they use and manage presets across versions. A function based implementation would in such cases be more flexible and efficient than a class hierarchy based one.

subclassing opens up more than just categorization. For example, each preset could accept different judge models (e.g, Agent(model="openai:/gpt-4o")), different inference parameters, or other preset-specific configurations.

A function based implementation does not prevent such customizations.

[Nehanth]

Sorry for the delay in addressing comments.

Thanks for the feedback @mprahl @B-Step62 @etirelli. I updated the RFC to address the concern around this being a burden to maintain.

As @B-Step62 suggested, I dropped the Safety and Quality presets and kept Rag, Agent, and ConversationalAgent as starting points for new users. I also added customization and persistence so teams can create custom presets, save them to the MLflow server, and share them across sessions. This addresses the version stability concern since persisted presets are user-controlled and don't change on upgrade. Please take a look.

[jwm4]

This RFC is on the right track. The core idea of bundling scorers into named, shareable presets is solid.

My main feedback is that the persistence story needs more user-facing specification (see my comment on that section), and that the RFC still leans heavily into implementation detail (private method bodies, deduplication internals, or/ror implementations). An RFC should nail down the requirements and the user-facing API. Implementation choices like how deduplication is coded or how subclass constructors work are better left to the PR.

[jwm4]

Persistence looks very useful to me. The API surface (register() / get_preset()) is clear, but a few user-facing behavioral questions should be answered at the RFC level:

1. Custom scorer portability. The example shows my_custom_scorer in a persisted preset. When a teammate calls get_preset("my_team_eval"), what do they get back? Do they need the same custom scorer code available in their environment? What happens if they don't? This is central to the team-sharing story.
2. Scope/namespace. The experiment_id parameter suggests presets are scoped to experiments, but the default behavior (no experiment_id) isn't specified. Are presets workspace-global by default? If I register to experiment A, can someone working in experiment B see it?
3. Discovery. There's get_preset(name=...) for retrieving a known preset, but no way to ask "what presets are available here?" Something like list_presets() seems like a natural companion, especially for the team-sharing use case.

These don't need implementation details, just a description of the intended user experience.

[jwm4]

This implementation reassigns self._scorers after super().init() has already run _validate_no_duplicates, so the added scorers bypass the duplicate check. Consider having the constructor accept extra scorers so the full list goes through validation, e.g.:

class MyAgent(Agent):
    def __init__(self):
        super().__init__(extra_scorers=[Fluency(), my_compliance_scorer])

Alternatively, this level of implementation detail could be dropped from the RFC entirely. The RFC really should be focused more on the specification and not the code.

[jwm4]

_deduplicate returns a plain list, so Agent() | Rag() gives you a list, not a Preset. That means you can't chain (Agent() | Rag() | [my_scorer]), can't call register() on the result, and the return type silently changes from Preset to list. This should probably return a new Preset.

Alternatively, this level of implementation detail could be dropped from the RFC, with just the requirement that | combines presets with deduplication and produces a Preset.