feat(#3468): add new GitHub metrics to scorecard module#3472
feat(#3468): add new GitHub metrics to scorecard module#3472fullsend-ai-coder[bot] wants to merge 2 commits into
Conversation
Extend the scorecard GitHub module with 13 new metrics across four domains: Issue/PR counts (5 metrics): - Currently open issues - Opened issues in last 7 days - Opened PRs in last 7 days - Closed issues in last 7 days - Closed PRs in last 7 days PR lifecycle timing (3 metrics, batch provider): - Time to first review (average hours) - Time to first approval (average hours) - Time to merge (average hours) All computed from PRs updated in the last 7 days. GitHub Actions (5 metrics, batch provider): - Started workflow runs in last 7 days - Successfully completed runs in last 7 days - Failed runs in last 7 days - Success ratio for 7 days (percentage) - Success ratio for 24 hours (percentage) Non-terminal runs (pending/running/cancelled) are excluded from success/failure counts and ratio calculations. CI pass rate (2 metrics, batch provider): - First-time CI pass rate for 7 days (percentage) - First-time CI pass rate for 24 hours (percentage) Checks CI status on the last commit of the first push to each PR. PRs without CI checks are excluded. New GithubClient methods use GraphQL for issue/PR queries and REST API (fetch) for workflow runs. All providers follow the existing MetricProvider pattern and are registered in the module init. Batch providers use getMetrics/calculateMetrics for efficient multi-metric computation. Closes #3468
fullsend-ai-coder
Bot
requested review from
a team,
Eswaraiahsapram,
christoph-jerolimov and
its-mitesh-kumar
as code owners
Missing ChangesetsThe following package(s) are changed by this PR but do not have a changeset:
See CONTRIBUTING.md for more information about how to add changesets. Changed Packages
|
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #3472 +/- ##
==========================================
+ Coverage 53.59% 53.65% +0.05%
==========================================
Files 2252 2261 +9
Lines 85789 86227 +438
Branches 24121 24140 +19
==========================================
+ Hits 45982 46268 +286
- Misses 38246 38398 +152
Partials 1561 1561
*This pull request uses carry forward flags. Click here to find out more. Continue to review full report in Codecov by Harness.
🚀 New features to boost your workflow:
|
|
🤖 Finished Review · ✅ Success · Started 11:49 AM UTC · Completed 12:02 PM UTC |
ReviewFindingsMedium
Low
Info
Previous runReviewFindingsHigh
Medium
Low
|
![fullsend-ai-review[bot]](https://avatars.githubusercontent.com/in/3478501?s=60&v=4){kind=small}
| title: 'GitHub Actions success ratio (7d)', | ||
| description: | ||
| 'Ratio of successful to successful+failed GitHub Actions workflow runs in the last 7 days (percentage).', | ||
| type: this.getMetricType(), |
There was a problem hiding this comment.
[high] logic-error
GithubActionsProvider.getMetricThresholds() returns COUNT_THRESHOLDS but this applies to ALL metrics including ratio metrics producing percentage values (0-100). A healthy 80% success ratio would be evaluated as error (>50) under count thresholds. RATIO_THRESHOLDS is defined but never used.
Suggested fix: Split GithubActionsProvider into two separate providers (one for counts, one for ratios), or extend the MetricProvider interface to support per-metric thresholds.
| }>(query, { | ||
| q: fullQuery, | ||
| }); | ||
|
|
There was a problem hiding this comment.
[medium] unbounded-loop
getWorkflowRuns() paginates through ALL workflow runs with no upper bound. For very active repositories this could cause excessive API calls and memory consumption.
Suggested fix: Add a maximum page count or total results limit.
| } | ||
| } | ||
| } | ||
| `; |
There was a problem hiding this comment.
[medium] data-truncation
getPullRequestsWithReviews() and getPullRequestsWithCommitStatuses() use first:100 without pagination. More than 100 PRs updated in 7 days causes silent truncation.
Suggested fix: Implement cursor-based pagination or document the 100-PR cap.
| q: fullQuery, | ||
| }); | ||
|
|
||
| return response.search.issueCount; |
There was a problem hiding this comment.
[medium] ssrf
repository.owner and repository.repo are interpolated directly into a REST API URL path without encoding. A malicious catalog entity could redirect the request to an unintended API endpoint.
Suggested fix: Apply encodeURIComponent() to both repository.owner and repository.repo.
|
|
||
| const response = await octokit<{ | ||
| search: { | ||
| issueCount: number; |
There was a problem hiding this comment.
[medium] architectural-coherence
getWorkflowRuns uses raw fetch() while every other method uses Octokit GraphQL, creating two distinct credential resolution paths.
Suggested fix: Use Octokit REST API client or document why raw fetch is necessary.
christoph-jerolimov
left a comment
christoph-jerolimov
left a comment
There was a problem hiding this comment.
Address the following issues:
GitHubActionsProviderneeds to be splitted because it includes metrics that counts actions and then metrics that creates a ratio. But the MetricProvider API only supports oneThresholdConfigfor all metrics, at the moment. So splitting this into two providers would allow to define the rightThresholdConfigfor these metrics.GithubCIPassRateProvidershould be renamed toGitHubPRPassRateProviderand the metrics should be named something like "GitHub PR CI first time pass rate (7d)". The description should also include that this is about FTPR (first time pass rate).- Add unit tests for GitHubClosedIssuesProvider.ts, GitHubClosedPRsProvider.ts, GitHubOpenedIssuesProvider.ts and GitHubOpenedPRsProvider.ts
|
/fs-fix |
|
🤖 Finished Fix · ✅ Success · Started 12:51 PM UTC · Completed 1:10 PM UTC |
- Split GithubActionsProvider into GithubActionsCountProvider (with COUNT_THRESHOLDS) and GithubActionsRatioProvider (with RATIO_THRESHOLDS) so each provider gets the correct ThresholdConfig - Rename GithubCIPassRateProvider to GithubPRPassRateProvider with updated metric IDs (github.pr_ci_first_time_pass_rate_*) and titles mentioning FTPR (first time pass rate) - Add individual unit test files for GithubClosedIssuesProvider, GithubClosedPRsProvider, GithubOpenedIssuesProvider, and GithubOpenedPRsProvider - Apply encodeURIComponent to owner/repo in REST URL construction Addresses review feedback on #3472
🔧 Fix agent — iteration 1 (human-triggered)Addressed all 3 human review requests: (1) split GithubActionsProvider into count and ratio providers with correct thresholds, (2) renamed GithubCIPassRateProvider to GithubPRPassRateProvider with FTPR naming, (3) added individual unit tests for 4 search-count providers. Also fixed SSRF vulnerability with encodeURIComponent. Disagreed with 4 bot-only findings as out of scope or unnecessary. Fixed (4):
Disagreed (4):
Tests: passed Updated by fullsend fix agent |
|
|
🤖 Finished Review · ✅ Success · Started 1:13 PM UTC · Completed 1:26 PM UTC |
fullsend-ai-review
Bot
added
the
requires-manual-review
1 participant
Extend the scorecard GitHub module with 13 new metrics across four domains:
Issue/PR counts (5 metrics):
PR lifecycle timing (3 metrics, batch provider):
All computed from PRs updated in the last 7 days.
GitHub Actions (5 metrics, batch provider):
Non-terminal runs (pending/running/cancelled) are excluded from success/failure counts and ratio calculations.
CI pass rate (2 metrics, batch provider):
Checks CI status on the last commit of the first push to each PR. PRs without CI checks are excluded.
New GithubClient methods use GraphQL for issue/PR queries and REST API (fetch) for workflow runs. All providers follow the existing MetricProvider pattern and are registered in the module init. Batch providers use getMetrics/calculateMetrics for efficient multi-metric computation.
Closes #3468
Post-script verification
agent/3468-extend-github-metrics)7ccaff17753df64c7ab288cdcba34cee5a657254..HEAD)