Skip to content

feat: add agent scorer authoring#317

Open
nabilnaffar-sf wants to merge 2 commits into
forcedotcom:mainfrom
nabilnaffar-sf:nnaffar/scorers
Open

feat: add agent scorer authoring#317
nabilnaffar-sf wants to merge 2 commits into
forcedotcom:mainfrom
nabilnaffar-sf:nnaffar/scorers

Conversation

@nabilnaffar-sf

Copy link
Copy Markdown

What does this PR do?

Add core logic for generating the AiAgentScorerDefinition metadata

What issues does this PR fix or reference?

new feature

@nabilnaffar-sf
nabilnaffar-sf marked this pull request as ready for review July 7, 2026 06:02
@jeniok

jeniok commented Jul 12, 2026

Copy link
Copy Markdown

Nice work getting this out — the shape is right and the XML matches what the org expects. A few things I'd like to see addressed before we approve; the top three are library-level correctness issues that let
invalid specs slip through to on-disk metadata.

Blocking

  1. validateScorerSpec doesn't require specification for dataType: Number — src/agentScorer.ts:135-147

validateScorerSpec({ dataType: 'Number', ...no specification... }) → no throw

buildScorerXml then produces a Number scorer with zero entries and no block. Add:

if (spec.dataType === 'Number' && !spec.specification) {
throw new Error("specification is required when dataType is 'Number'.");
}

  1. validateScorerSpec doesn't validate lightningType against SUPPORTED_LIGHTNING_TYPES — src/agentScorer.ts:149-151

validateScorerSpec({ dataType: 'LightningType', lightningType: 'bogus__type', ... }) → no throw

The CLI's JSON schema catches this today, but library-only callers get no protection. Please cross-check against the const you already export.

  1. buildScorerXml silently drops user-supplied outputEnumValues when dataType === 'Number' — src/agentScorer.ts:219-243

The Number branch always regenerates enum values from specification and ignores anything the caller passed in outputEnumValues. Either reject the combination in validateScorerSpec (mutually exclusive) or honor
the caller's values.

Should fix

  1. No validation that Text scorers have outputEnumValues at all. validateScorerSpec only runs the fallback-count check when outputEnumValues is defined. A Text scorer with no output values passes validation and
    produces XML with no outcomes.

  2. No validation of agentAssociation.samplingRate ∈ [0, 1]. The CLI's YAML schema enforces this; library callers get no bounds check.

  3. buildPromptTemplateXml hard-codes primaryModel: 'sfdc_ai__DefaultOpenAIGPT4OmniMini' — src/agentScorer.ts:333. If intentional for v1, drop a short // WHY comment; otherwise expose as an optional spec field.

  4. Duplicated mkdir for promptDir — src/agentScorer.ts:377 and :385 both compute join(options.outputDir, 'genAiPromptTemplates'). Harmless (recursive: true) but the variable gets redeclared.

Nice-to-have

  1. No unit tests in this PR. All coverage currently lives downstream in plugin-agent PR #456, and that suite is broken. A small test/agentScorer.test.ts here would let the library stand on its own.

  2. String(rounded) for number enum values strips trailing zeros — src/agentScorer.ts:110-115. Step 0.1 produces '0', '0.1', '0.5', '1'. Please confirm the backend accepts trailing-zero-stripped forms; if it
    wants "0.10" or "1.0", this drifts.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants