feat(rules): refactor code to rename evaluators to rules

[namrataghadi-galileo]

Summary

• What changed and why.

Scope

• User-facing/API changes:
• Internal changes:
• Out of scope:

Risk and Rollout

• Risk level: low / medium / high
• Rollback plan:

Testing

• Added or updated automated tests
• Ran make check (or explained why not)
• Manually verified behavior

Checklist

• Linked issue/spec (if applicable)
• Updated docs/examples for user-facing changes
• Included any required follow-up tasks

[codecov]

Codecov Report

❌ Patch coverage is 97.87234% with 9 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
rules/builtin/src/agent_control_rules/json/rule.py	89.65%	3 Missing ⚠️
rules/builtin/src/agent_control_rules/sql/rule.py	91.66%	3 Missing ⚠️
models/src/agent_control_models/controls.py	96.87%	1 Missing ⚠️
...ules/builtin/src/agent_control_rules/_discovery.py	98.43%	1 Missing ⚠️
...co/src/agent_control_rule_cisco/ai_defense/rule.py	91.66%	1 Missing ⚠️

📢 Thoughts on this report? Let us know!

[lan17]

Why are we doing this?

Evaluator is a good name, no?

[namrataghadi-galileo]

@lan17 This is to comply with Cisco/Splunks naming as ACE is offered now as a product within Splunk Cloud Observability. The keyword "evaluators" is reserved for "metrics".. Heres the proposal we are going to go ahead with after our internal discussions. Credits to @abhinav-galileo for coining this proposal.

{
  "scope": {
    "step_types": ["llm"],
    "stages": ["post"]
  },
  "condition": {
    "selector": { "path": "output" },
    "evaluator": {
      "name": "luna.toxicity",
      "config": {
        "operator": "gte",
        "threshold": 0.7
      }
    }
  },
  "action": { "decision": "deny" }
}

This PR is going to change. We will rename evaluators to checks and not rules.
So the terms are:

Evaluator = reusable compute/check primitive, e.g. Toxicity, Context Relevance, Regex, JSON (this complies with Splunk Observability nomenclature)
Provider = Luna, Azure Content Safety, Guardrails AI, Agent Control built-ins, etc.
Condition/check = selected input + evaluator + config/operator/threshold -> matched/not matched
Control = scope/stage + condition/check + action

In Agent Control, controls use evaluators through conditions/checks. The action is applied when the condition matches.

[lan17]

That's same as current design, no?

[namrataghadi-galileo]

@lan17 There is subtle difference in how it works today

ControlDefinition = scope/stage + condition + action
condition = selector + evaluator_spec, or a boolean tree of conditions
evaluator_spec = evaluator + config

Example today:
{
  "scope": {
    "step_types": ["llm"],
    "stages": ["post"]
  },
  "condition": {
    "selector": { "path": "output" },
    "evaluator": {
      "name": "galileo.luna",
      "config": {
        "scorer_label": "toxicity",
        "operator": "gte",
        "threshold": 0.7
      }
    }
  },
  "action": { "decision": "deny" }
}

Here, "galileo.luna" is the evaluator, while "toxicity" is hidden inside its config. The proposal is to bring out the metrics like toxicity out as evaluators like below

"evaluator": {
      "name": "luna.toxicity",
      "config": {
        "operator": "gte",
        "threshold": 0.7
      }
    }

I would even go further and change this to

"evaluator": {
      "name": "toxicity",
      "provider": "luna",
      "config": {
        "operator": "gte",
        "threshold": 0.7
      }
    }

[lan17]

I'm still confused since evaluators currently are regex, Luna, etc, no?

[namrataghadi-galileo]

“Evaluator” is a good name, but Splunk Observability already uses it for a reusable metric that returns a score, boolean, or findings. In Agent Control, today’s evaluator also applies thresholds or matching logic and decides whether a control triggers, so it is closer to a check.
For example: Luna evaluates toxicity and returns a score; “toxicity ≥ 0.7” is the check. Similarly, regex evaluates whether a pattern exists, while the check decides to trigger on a match; JSON/SQL evaluators return validation findings, while checks decide which findings trigger.
The proposed terminology is: Provider implements Evaluators; Checks turn evaluator outputs into matched/not matched; Conditions combine checks; and Controls add scope and action. This keeps Agent Control consistent with Splunk Observability and avoids “evaluators inside evaluators.”

{
  "scope": {
    "step_types": ["llm"],
    "stages": ["post"]
  },
  "condition": {
    "check": {
      "selector": {"path": "output"},
      "evaluator": {
        "provider": "galileo.luna",
        "name": "toxicity",
        "config": {
          "scorer_id": "..."
        }
      },
      "match": {
        "operator": "gte",
        "value": 0.7
      }
    }
  },
  "action": {
    "decision": "deny"
  }
}

[abhinav-galileo]

@namrataghadi-galileo - I am not sure about adding another level of nesting with check..