feat: Add generic BaseAdapter framework for third-party evaluator integration (DeepEval + Autoevals) by stone-coding · Pull Request #528 · aws/bedrock-agentcore-sdk-python

stone-coding · 2026-06-16T22:46:47Z

Issue P446281164 — Third-Party Evaluator Integration (Phase 1)

Description of changes Generic BaseAdapter framework that adapts 3P evaluation libraries into AgentCore-compatible Lambda handlers. Supports DeepEval, Autoevals, and is extensible for future libraries (RAGAS, etc.).

Key components:
BaseAdapter — shared orchestration: parse event (supports EvaluatorInput from @custom_code_based_evaluator() decorator) → extract fields → validate → execute with timeout → error handling
DeepEvalAdapter — runs any DeepEval BaseMetric. DeepEvalHandler alias for backward compat.
AutoevalsAdapter — runs any Autoevals scorer
Field extraction from _eval_log_records in ADOT spans (input, actual_output, retrieval_context, expected_output)
Thread-based timeout (default 290s)
field_mapper escape hatch for custom span extraction

Design decisions:
Composes with @custom_code_based_evaluator() decorator — accepts EvaluatorInput directly
Never raises unhandled exceptions — always returns valid response dict
Adding a new library = one ~20 line subclass with execute() method

Introduces a new integrations/deepeval/ module that adapts AgentCore Lambda evaluation events into DeepEval LLMTestCase objects, runs any BaseMetric, and returns structured score/label/explanation responses.

…leTurnParams deprecation

…d EvaluatorInput support

jariy17 · 2026-06-18T18:54:11Z

@@ -0,0 +1,5 @@
+"""DeepEval integration for AgentCore Evaluation."""
+
+from bedrock_agentcore.evaluation.integrations.deepeval.handler import DeepEvalHandler


Since this is using eval's custom code evaluator please but this under custom_code_based_evaluators.

jariy17 · 2026-06-18T18:55:16Z

+import threading
+from typing import Any, Callable, Dict, Optional
+
+from deepeval.metrics import BaseMetric


Please add an integ test for this. Look into tests_integ for examples.

jariy17 · 2026-06-18T19:23:18Z

+import threading
+from typing import Any, Callable, Dict, Optional
+
+from deepeval.metrics import BaseMetric


Also, let's add this in our pyproject as an optional dependency, so customer's know which deepeval version we support.

jariy17 · 2026-06-18T19:24:36Z

+
+
+@dataclass
+class ParsedEvaluationEvent:


Please use EvaluatorInput from our code_based_evaluator. No need to duplicate lambda logic.

jariy17 · 2026-06-25T21:38:39Z

+            Error: {"errorCode": str, "errorMessage": str}
+        """
+        try:
+            if isinstance(event, EvaluatorInput):


ParsedEvaluationEvent and EvaluatorInput look like they're doing the same job — both just turn the raw lambda event into a structured input. call even copies one into the other field-for-field. Is there a reason we need a second type instead of reusing EvaluatorInput?

Proposal: make it a requirement that customers place these adapters within the @code_based_evaluators decorator. That way the adapter stops owning input/output validation and the decorator does it instead. Keeps the adapter focused on just running the eval.

jariy17 · 2026-06-25T22:27:10Z

+}
+
+
+def _get_required_params(metric: BaseMetric) -> List[str]:


metric.measure() already calls check_llm_test_case_params with the metric's own _required_params and raises MissingTestCaseParamsError.

So we can drop the registry: build the LLMTestCase with whatever fields we have, call measure(), and catch that error.

By doing this, we let customers use GEval too — its required fields aren't fixed on the class, they're whatever the customer passes to evaluation_params at construction, so a static registry can never cover it. Letting the metric validate itself handles that case for free.

jariy17 · 2026-06-25T22:37:11Z

+) -> Dict[str, Any]:
+    """Extract evaluation fields from AgentCore session spans.
+
+    Parses _eval_log_records from span attributes, filters by target_trace_id,


Can you tell me what otel agent semantic you are following here? Because I haven't seen any agent SDK emit this _eval_log_records?

jariy17 · 2026-06-25T22:54:26Z

+        self.validate_fields(fields)
+        return fields
+
+    def validate_fields(self, fields: Dict[str, Any]) -> None:


Can you add @AbstractMethod here please? The no-op default means a subclass that forgets to override it silently skips validation, and bad fields fail deeper in execute instead. Both adapters override it anyway, so abstract just makes each one declare its required fields on purpose.

jariy17 · 2026-06-25T23:06:22Z

+
+        thread = threading.Thread(target=target, daemon=True)
+        thread.start()
+        thread.join(timeout=self.timeout)


When the thread "times out" here, it doesn't actually end join just returns back to the caller while the worker keeps running. So if Lambda reuses the same container, we can have a background thread from a previous invocation still executing during the next one. I've heard this is a real failure case, so let's drop the thread machinery and let the AWS Lambda timeout handle it for us instead.

jariy17 · 2026-06-25T23:09:39Z

+
+    def __init__(
+        self,
+        field_mapper: Optional[Callable[[Dict[str, Any]], Dict[str, Any]]] = None,


Can we make extract_fields_from_spans the default value of field_mapper in the constructor? Then we have one extraction path instead of the if-field_mapper-else branch.

haomiao037 added 9 commits June 15, 2026 10:39

Add DeepEvalHandler integration with unit tests

ba80889

Introduces a new integrations/deepeval/ module that adapts AgentCore Lambda evaluation events into DeepEval LLMTestCase objects, runs any BaseMetric, and returns structured score/label/explanation responses.

Fix span extraction to use real AgentCore _eval_log_records structure

b0d9682

Set context field from tool messages for HallucinationMetric support

81a46dd

Use metric.success for label instead of manual threshold comparison

3080e40

Add model override and timeout enforcement to DeepEvalHandler

34674bb

Add model override, timeout enforcement, use metric.success, fix Sing…

6aedcbf

…leTurnParams deprecation

Fix _get_required_params to handle GEval unmappable typing params

2260eb3

Add .deepeval/ to gitignore

14f0354

Move model override to init to avoid per-call mutation

b109a64

stone-coding requested a review from a team June 16, 2026 22:46

stone-coding requested a deployment to manual-approval June 16, 2026 22:47 — with GitHub Actions Waiting

jariy17 mentioned this pull request Jun 18, 2026

Expose evaluationReferenceInputs (ground truth) on EvaluatorInput for code-based evaluators #539

Closed

Refactor to BaseAdapter framework with DeepEval/Autoevals adapters an…

4e74926

…d EvaluatorInput support

stone-coding requested a deployment to manual-approval June 24, 2026 23:26 — with GitHub Actions Waiting

stone-coding changed the title ~~feat: Add DeepEvalHandler for third-party evaluator integration~~ feat: Add generic BaseAdapter framework for third-party evaluator integration (DeepEval + Autoevals) Jun 25, 2026

jariy17 requested changes Jun 25, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat: Add generic BaseAdapter framework for third-party evaluator integration (DeepEval + Autoevals)#528

feat: Add generic BaseAdapter framework for third-party evaluator integration (DeepEval + Autoevals)#528
stone-coding wants to merge 10 commits into
aws:mainfrom
stone-coding:deepeval-handler

stone-coding commented Jun 16, 2026 •

edited

Loading

Uh oh!

jariy17 Jun 18, 2026

Uh oh!

jariy17 Jun 18, 2026

Uh oh!

jariy17 Jun 18, 2026

Uh oh!

jariy17 Jun 18, 2026

Uh oh!

jariy17 Jun 25, 2026

Uh oh!

jariy17 Jun 25, 2026

Uh oh!

jariy17 Jun 25, 2026

Uh oh!

jariy17 Jun 25, 2026

Uh oh!

jariy17 Jun 25, 2026

Uh oh!

jariy17 Jun 25, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

		@@ -0,0 +1,5 @@
		"""DeepEval integration for AgentCore Evaluation."""

		from bedrock_agentcore.evaluation.integrations.deepeval.handler import DeepEvalHandler



		@dataclass
		class ParsedEvaluationEvent:

Uh oh!

Conversation

stone-coding commented Jun 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

stone-coding commented Jun 16, 2026 •

edited

Loading