Skip to content

Instrument tool calling to track tool invocations, arguments, and results #474

@psschwei

Description

@psschwei

Description:
Instrument tool calling to track tool invocations, arguments, and results.

Detailed Requirements:

  1. Add spans for tool operations:
    • tool.call - Tool invocation
  2. Add attributes:
    • tool.name - Name of the tool being called
    • tool.arguments_hash - Hash of arguments (for privacy)
    • tool.status - success/error
  3. Add span events for:
    • Tool call start with argument summary (redacted)
    • Tool call completion with result summary
    • Tool call error with exception details
  4. Handle tool call extraction from model responses
  5. Track multiple tool calls in single response

Files to Modify:

  • mellea/stdlib/tools/interpreter.py - Tool execution instrumentation
  • mellea/backends/tools.py - Tool call extraction
  • mellea/helpers/openai_compatible_helpers.py - OpenAI tool handling

Span Hierarchy Example:

backend.request
├── [model returns tool calls]
└── tool.call (tool_1)
    └── [tool execution]
└── tool.call (tool_2)
    └── [tool execution]

Acceptance Criteria:

  • Each tool invocation has a span
  • Tool name and status captured
  • Arguments handled with privacy in mind
  • Multiple tool calls tracked separately

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions