Skip to content

Instrument the sampling loop to track each iteration attempt, providing visibility into retry behavior #472

@psschwei

Description

@psschwei

Description:
Instrument the sampling loop to track each iteration attempt, providing visibility into retry behavior.

Detailed Requirements:

  1. Add spans for sampling operations:
    • sampler.sample - Overall sampling operation
    • sampler.iterate - Single sampling iteration
    • sampler.repair - Repair attempt
    • sampler.select - Selection from candidates
  2. Add attributes:
    • strategy - Sampling strategy name
    • iteration - Current iteration number
    • max_iterations - Maximum allowed iterations
    • status - success/failure/repair
  3. Track iteration outcomes:
    • Add span events for validation pass/fail
    • Record reason for rejection

Files to Modify:

  • mellea/stdlib/sampling/base.py - Instrument BaseSamplingStrategy
  • mellea/stdlib/sampling/rejection.py - Instrument RejectionSamplingStrategy
  • mellea/stdlib/sampling/sofai.py - Instrument SoFAISamplingStrategy
  • mellea/stdlib/sampling/majority_voting.py - Instrument MajorityVotingStrategy

Span Hierarchy Example:

sampler.sample
├── sampler.iterate (attempt 1)
│   ├── backend.request
│   └── validator.check (FAIL)
├── sampler.iterate (attempt 2)
│   ├── backend.request
│   └── validator.check (PASS)
└── [success]

Acceptance Criteria:

  • Each sampling iteration has its own span
  • Iteration count and outcomes tracked
  • Strategy-specific behavior captured
  • Spans correctly nested

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions