fix(.github/scorecard): calibrate scoring strictness and fix utility overall math#1013
Open
bpmct wants to merge 4 commits into
Open
fix(.github/scorecard): calibrate scoring strictness and fix utility overall math#1013bpmct wants to merge 4 commits into
bpmct wants to merge 4 commits into
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to #1012, which merged before the last two commits on its branch landed. This carries the scoring calibration fix and the index intro restructure.
Why
An audit of the initial batch found the model systematically over-crediting Restricted-Network Readiness. 10 modules scored 20/20; most were wrong. The model's own notes rationalized credit with reasoning like "the URL could be proxied/mirrored at network level" (any URL can) and endpoints being "implicit in API key usage" (implicit is not documented). claude-code scored 20/20 on restricted-network with no mirror variable and no air-gap docs.
Changes
SCORECARD.md: tightened criteria definitions. Mirrorable artifact source requires an actual module variable overriding the tool's own download URL (auxiliary config URLs, proxies, and DNS rewrites do not count). Egress transparency requires a dedicated section enumerating endpoints. New Strictness rule: never credit out-of-module workarounds or infer undocumented behavior.score-modules.ts: calibration anchors in the scoring prompt. Highlights: name the exact URL-override variable or score 0; offline/skip-install modes count only toward bring-your-own binary; theme-level N/A is for modules that download nothing, 0 is for missing capability; inline placeholder keys in README examples cap the secrets criterion at half; perfect theme scores trigger a self-recheck.score-modules.ts:fixOverallnow catches a utility overall emitted asX / 75and normalizes it to/ 100(previously passed through unrecomputed).scorecard-index.ts: index intro restructured as what / why / how it's generated.Validation
All 48 discussions were re-scored with this code. Only 3 modules retain 20/20 restricted-network, each verified legitimate: Amazon Q (
q_install_url+ air-gapped section), Portable Desktop (url+ checksum), Mux (registry_url). claude-code dropped to 67 (matches a manual hand-score of ~65), code-server 97 → 87. Live results: 📊 Module Scorecards.Known limitation, unchanged by this PR: ±3-5 run-to-run noise on subjective criteria even at temperature 0. If the PR regression comments get noisy, a follow-up can add a tolerance band.
🤖 Generated with Coder Agents