Skip to content

fix(eval): trim legacy trajectory span history#1630

Open
Chibionos wants to merge 1 commit into
mainfrom
fix/ae-1482-trim-legacy-trajectory
Open

fix(eval): trim legacy trajectory span history#1630
Chibionos wants to merge 1 commit into
mainfrom
fix/ae-1482-trim-legacy-trajectory

Conversation

@Chibionos
Copy link
Copy Markdown
Contributor

Summary

  • Fixes AE-1482 in the uipath-python evaluator layer by preventing the legacy trajectory evaluator from serializing full raw span attributes into the judge prompt.
  • Reuses the compact tool-history formatter already used by LLMJudgeTrajectoryEvaluator, keeping tool name, arguments, and output while dropping prompt-bearing LLM spans and arbitrary span metadata.
  • Adds a regression test with prompt-sized LLM/tool metadata to ensure AgentRunHistory does not reintroduce the duplicated system prompt.

Context

The reported trace bloat came from the legacy evaluator copying entire OpenTelemetry span payloads into the evaluation prompt. Raw LLM spans can legitimately contain growing message history, but the legacy trajectory judge does not need that full payload; it needs the compact tool trajectory.

This does not implement the separate sub-agent/deeper-trace trimming toggle discussed for trajectory scope. It addresses the immediate duplication/bloat bug in the legacy evaluator prompt construction.

Validation

  • uv run pytest tests/evaluators/test_legacy_trajectory_evaluator.py tests/evaluators/test_evaluator_methods.py -k trajectory -q
  • uv run ruff check src/uipath/eval/evaluators/legacy_trajectory_evaluator.py tests/evaluators/test_legacy_trajectory_evaluator.py
  • uv run ruff format --check src/uipath/eval/evaluators/legacy_trajectory_evaluator.py tests/evaluators/test_legacy_trajectory_evaluator.py
  • git diff --check

@github-actions github-actions Bot added test:uipath-langchain Triggers tests in the uipath-langchain-python repository test:uipath-integrations labels May 14, 2026
@sonarqubecloud
Copy link
Copy Markdown

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

test:uipath-integrations test:uipath-langchain Triggers tests in the uipath-langchain-python repository

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants