Normalize safe output outcome evaluation and demote existence-only fallback

[Copilot]

This change introduces a shared normalized outcome model for safe output evaluation and updates the existence-only fallback to report weak unknown evidence instead of accepted. It also propagates the normalized fields through JSONL/telemetry output and adds summary buckets so dashboards can distinguish strong outcomes from fallback-only signals.

• Shared outcome model

  • Adds OutcomeStatus with: accepted, rejected, pending, ignored, skipped, unknown
  • Adds EvidenceStrength with: strong, medium, weak
  • Adds shared OutcomeEvaluation fields:
    • outcome_status
    • evidence_strength
    • signal

• Fallback semantics

  • Changes the generic existence-only evaluator from implicit acceptance to:
    • outcome_status: unknown
    • evidence_strength: weak
    • signal: target_exists_only
  • Prevents existence-only fallbacks from inflating accepted metrics

• Reporting and telemetry

  • Emits normalized fields in outcome JSONL
  • Carries normalized fields into OTLP outcome spans
  • Keeps legacy result alongside normalized fields for compatibility where needed

• Summary / dashboard fields

  • Adds evidence-strength acceptance buckets:
    • accepted_strong
    • accepted_medium
    • accepted_weak
  • Adds fallback tracking:
    • fallback_exists_only_count

• Coverage

  • Adds focused Go tests for:
    • normalized outcome derivation
    • existence-only fallback behavior
    • summary exclusion of fallback-only outcomes from accepted counts
    • JSONL emission of normalized fields
  • Adds focused JS tests for normalization and OTLP span export fields

Example of the normalized payload shape:

{
  "result": "unknown",
  "outcome_status": "unknown",
  "evidence_strength": "weak",
  "signal": "target_exists_only",
  "detail": "object still exists"
}

---

pr-sous-chef: updated branch on https://github.com/github/gh-aw/actions/runs/26481536311

Generated by 👨‍🍳 PR Sous Chef · gpt54 17.2M · ◷

---

pr-sous-chef: branch updated during run https://github.com/github/gh-aw/actions/runs/26484943570

Generated by 👨‍🍳 PR Sous Chef · gpt54 7.1M · ◷

[Copilot]

Pull request overview

This PR introduces a shared, normalized outcome model for safe output evaluation (status + evidence strength + signal), changes the existence-only fallback to report weak/unknown instead of accepted, and propagates the normalized fields through JSONL/telemetry outputs so dashboards can distinguish strong outcomes from fallback-only signals.

Changes:

• Add normalized outcome fields (outcome_status, evidence_strength, signal) and derive them from legacy result/detail when needed.
• Demote existence-only fallback from accepted to weak/unknown and track it explicitly in summaries.
• Emit normalized fields in JSONL and include them in OTLP spans/summary attributes, with tests covering the new behavior.

Show a summary per file

File	Description
pkg/cli/outcome_evaluation.go	Adds normalized outcome model types and normalization logic.
pkg/cli/outcome_eval.go	Embeds normalized fields in reports; updates summary aggregation to use normalized status/strength and adds new buckets.
pkg/cli/outcome_eval_test.go	Adds Go tests for normalization, existence-only fallback behavior, summary bucketing, and JSONL emission.
pkg/cli/outcome_eval_jsonl.go	Emits normalized fields in JSONL and adds write error handling.
pkg/cli/outcome_eval_generic.go	Changes generic existence-only fallback to weak/unknown and makes GH API getter injectable for tests.
actions/setup/js/evaluate_outcomes.cjs	Adds JS normalization helper and propagates normalized fields into JSONL + fleet summary counters.
actions/setup/js/evaluate_outcomes.test.cjs	Adds JS tests for normalization behavior.
actions/setup/js/emit_outcome_spans.cjs	Adds normalized span attributes + status mapping and summary attributes for new buckets.
actions/setup/js/emit_outcome_spans.test.cjs	Updates span export tests for normalized attributes and new summary fields.

Copilot's findings

Tip

Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

• Files reviewed: 9/9 changed files
• Comments generated: 2

[github-actions]

Please resolve the unstable merge/check state so Copilot can keep this moving.

pr-sous-chef: https://github.com/github/gh-aw/actions/runs/26478924995

Generated by 👨‍🍳 PR Sous Chef · gpt54 33.6M · ◷

[github-actions]

@copilot review all comments and address the unresolved review feedback.

Generated by 👨‍🍳 PR Sous Chef · gpt54 7.1M · ◷

[pelikhan]

@copilot resolve the merge conflicts in this pull request

[Copilot]

@copilot resolve the merge conflicts in this pull request

Resolved. The only conflict was in pkg/cli/outcome_eval.go (OutcomeSummary struct — the new bucket fields from this branch conflicted with main's version). Kept all new fields (AcceptedStrong, AcceptedMedium, AcceptedWeak, FallbackExistsOnlyCount, CostPerAcceptedOutcome) and also fixed a stray extra argument in a test call that was left from development. All CLI tests pass.