Skip to content

Add type-specific outcome evaluators for create_issue, add_comment, and add_labels#35113

Draft
Copilot wants to merge 4 commits into
mainfrom
copilot/implement-outcome-evaluators-again
Draft

Add type-specific outcome evaluators for create_issue, add_comment, and add_labels#35113
Copilot wants to merge 4 commits into
mainfrom
copilot/implement-outcome-evaluators-again

Conversation

Copy link
Copy Markdown
Contributor

Copilot AI commented May 27, 2026

This PR replaces existence-based outcome classification for create_issue, add_comment, and add_labels with dedicated evaluators that use engagement/lifecycle signals and persisted label state. It also adds branch-complete coverage for accepted/rejected/pending/unknown outcomes across these types.

  • Outcome evaluation: dedicated classifiers

    • actions/setup/js/evaluate_outcomes.cjs now routes create_issue, add_comment, and add_labels through type-specific evaluators.
    • create_issue now classifies based on engagement and closure behavior:
      • accepted (strong): open + merged PR/commit/closing reference
      • accepted (medium): open + non-author comment/reaction
      • rejected (strong): immediate close by non-author (windowed)
      • rejected (medium): closed with no activity
      • pending/unknown fallbacks
    • add_comment now classifies based on comment lifecycle and thread activity:
      • rejected (strong): comment deleted
      • accepted (strong): reacted/replied/quoted
      • accepted (medium): later thread activity
      • pending/unknown fallbacks
    • add_labels now evaluates retention using before-state + added labels with a retention window:
      • accepted (strong): added labels retained
      • rejected (strong): removed by non-author within retention logic
      • pending before window elapses
      • unknown when before-state is unavailable
  • Manifest/data model updates for add_labels

    • actions/setup/js/add_labels.cjs captures labelsBefore before label mutation.
    • actions/setup/js/safe_output_manifest.cjs persists labelsBefore and labelsAdded for downstream evaluation.
  • Focused test coverage for outcome branches

    • Added actions/setup/js/evaluate_outcomes.test.cjs to cover outcome branches for all three evaluators.
    • Updated actions/setup/js/safe_output_manifest.test.cjs to assert new label metadata extraction.
if (item.type === "create_issue") {
  return evaluateCreateIssue(item, itemRepo, timestamp, out, apiGet, nowMs);
}
if (item.type === "add_comment") {
  return evaluateAddComment(item, itemRepo, timestamp, out, apiGet, nowMs);
}
if (item.type === "add_labels") {
  return evaluateAddLabels(item, itemRepo, timestamp, out, apiGet, nowMs);
}

Copilot AI and others added 3 commits May 27, 2026 04:13
Co-authored-by: gh-aw-bot <259018956+gh-aw-bot@users.noreply.github.com>
Co-authored-by: gh-aw-bot <259018956+gh-aw-bot@users.noreply.github.com>
Co-authored-by: gh-aw-bot <259018956+gh-aw-bot@users.noreply.github.com>
Copilot AI changed the title [WIP] Implement dedicated outcome evaluators for create_issue, add_comment, and add_labels Add type-specific outcome evaluators for create_issue, add_comment, and add_labels May 27, 2026
Copilot AI requested a review from gh-aw-bot May 27, 2026 04:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[plan] Implement dedicated outcome evaluators for create_issue, add_comment, and add_labels

2 participants