diff --git a/.github/scorecard/SCORECARD.md b/.github/scorecard/SCORECARD.md index 584a759a2..8115989e8 100644 --- a/.github/scorecard/SCORECARD.md +++ b/.github/scorecard/SCORECARD.md @@ -8,22 +8,23 @@ Score each criterion as **0 / half / full**. ### Universal criteria — 75 pts -| Criterion | Pts | Pass = | -| -------------------------------------------------- | -----: | ---------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| **Presentation & Onboarding** | **25** | | -| Configuration-mode examples | 12 | If the module has many options, each major mode has a documented example with sensible defaults, for example provider choice, app vs headless | -| Coder-context framing | 8 | Explains what the module adds on top of Coder, names both Coder and the target tool, and shows where Coder fits in the flow | -| Visual preview | 5 | README includes an image, GIF, or video of the module in action | -| **Credential Hygiene** | **20** | | -| Secrets marked sensitive | 16 | Sensitive inputs are marked `sensitive = true`, and README examples avoid inline secrets | -| Non-hardcoded auth path | 4 | If possible, README shows a path that avoids pasting raw keys into templates, for example ServiceAccount, IAM/OAuth/external auth, API key helper, or AI Gateway | -| **Restricted-Network Readiness** _(if applicable)_ | **20** | | -| Mirrorable artifact source | 10 | Download or install URL is configurable so it can point to an internal mirror or artifact store instead of the public internet | -| Bring-your-own binary | 6 | Download or install can be disabled entirely when the tool is already baked into the image | -| Egress transparency | 4 | External endpoints are documented, plus notes for restricted or air-gapped environments | -| **Engineering Quality** | **10** | | -| Input quality | 6 | Inputs have clear descriptions, sensible defaults, and `validation` where appropriate | -| Test coverage | 4 | Clear testing story, `.tftest.hcl` primarily covers business logic, TypeScript tests cover end-to-end behavior | +| Criterion | Pts | Pass = | +| ------------------------------------------------------ | -----: | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| **Presentation & Onboarding** | **25** | | +| Configuration-mode examples | 12 | If the module has many options, each major mode has a documented example with sensible defaults, for example provider choice, app vs headless | +| Coder-context framing | 8 | Explains what the module adds on top of Coder, names both Coder and the target tool, and shows where Coder fits in the flow | +| Visual preview | 5 | README includes an image, GIF, or video of the module in action | +| **Credential Hygiene** | **20** | | +| Secrets marked sensitive | 16 | Sensitive inputs are marked `sensitive = true`, and README examples avoid inline secrets | +| Non-hardcoded auth path | 4 | If possible, README shows a path that avoids pasting raw keys into templates, for example ServiceAccount, IAM/OAuth/external auth, API key helper, or AI Gateway | +| **Restricted-Environment Readiness** _(if applicable)_ | **20** | | +| Mirrorable artifact source | 10 | A module input variable overrides the URL the tool itself is downloaded or installed from, so it can point to an internal mirror or artifact store. Variables for auxiliary config URLs do not count, and neither do workarounds outside the module such as transparent proxies or DNS rewrites | +| Bring-your-own binary | 5 | A documented way to disable download or install entirely when the tool is already baked into the image | +| Egress transparency | 3 | A dedicated README section enumerates the external endpoints contacted at install and runtime, with notes for restricted or air-gapped environments. Mentions scattered across unrelated examples earn at most half; inferable endpoints do not count | +| Runs without sudo | 2 | Install and runtime scripts work as an unprivileged user, or the README documents the no-sudo path. Scripts that require root for core functionality score 0; sudo needed only for optional features with a working fallback earns half | +| **Engineering Quality** | **10** | | +| Input quality | 6 | Inputs have clear descriptions, sensible defaults, and `validation` where appropriate | +| Test coverage | 4 | Clear testing story, `.tftest.hcl` primarily covers business logic, TypeScript tests cover end-to-end behavior | ### Agent track — 25 pts diff --git a/.github/scorecard/score-modules.ts b/.github/scorecard/score-modules.ts index f9b8b21a1..59f9b67a4 100644 --- a/.github/scorecard/score-modules.ts +++ b/.github/scorecard/score-modules.ts @@ -119,11 +119,19 @@ async function gatherModuleContext(moduleName: string): Promise { function fixOverall(scorecard: string): string { // Find the summary table's data row: first row whose cells are bold - // "**a / b**" fractions (or N/A), ending with the overall cell. + // "**a / b**" fractions (or N/A), ending with the overall cell. The + // overall cell may use any denominator (e.g. "67 / 75" from a utility + // module); it is recomputed and rewritten as "/ 100" regardless. const lines = scorecard.split("\n"); - const rowIdx = lines.findIndex( - (l) => /^\|\s*\*\*\d/.test(l.trim()) && l.includes("/ 100"), - ); + const rowIdx = lines.findIndex((l) => { + const t = l.trim(); + if (!/^\|\s*(\*\*\d|N\/A)/i.test(t)) return false; + const cells = t.split("|").filter((c) => c.trim() !== ""); + return ( + cells.length >= 3 && + cells.every((c) => /n\/a/i.test(c) || /\d+\s*\/\s*\d+/.test(c)) + ); + }); if (rowIdx === -1) return scorecard; const cells = lines[rowIdx] .split("|") @@ -145,7 +153,7 @@ function fixOverall(scorecard: string): string { lines[rowIdx] = `| ${cells.join(" | ")} |`; let out = lines.join("\n"); out = out.replace( - /#### Overall — \d+(?:\.\d+)? \/ 100/g, + /#### Overall — \d+(?:\.\d+)? \/ \d+/g, `#### Overall — ${overall} / 100`, ); // Rewrite or append the normalization line under the Overall heading. @@ -179,13 +187,30 @@ ${context} Score the module strictly against the rubric. Rules: - First decide the track: Agent, IDE, or Utility. Agent = AI coding agents (CLI agents like Claude Code, Goose, Aider). IDE = editors/IDEs users open (code-server, JetBrains, VS Code). Utility = everything else (auth, regions, git helpers, file browsers, etc). - Score each criterion 0, half, or full. Full = implemented AND documented. Half = partial, awkward, or under-documented. Zero = absent. -- Apply "if applicable" exclusions per the rubric: when a concern does not exist by construction, mark the criterion N/A, exclude its points from the denominator, and normalize the final score to 100. +- BE STRICT. When evidence is ambiguous or missing, score lower. Never rationalize credit. +- Never credit workarounds achievable outside the module. "The URL could be proxied or mirrored at the network level" is NOT a mirrorable artifact source; that criterion requires an actual module input variable that overrides the download/install URL. A hardcoded URL scores 0 regardless of what admins could do around it. +- "Documented" means an explicit README section or example. Endpoints or behavior that are merely implicit, inferable, or visible only in source code do not count as documented. +- Apply "if applicable" exclusions per the rubric: when a concern does not exist by construction, mark the criterion N/A, exclude its points from the denominator, and normalize the final score to 100. Do not use N/A for concerns that exist but are unaddressed. - Utility modules skip the track section (denominator 75 before any N/A exclusions), then normalize: round(raw / denominator * 100). - Notes must be specific and evidence-based (reference actual variables, README sections, or files you saw). Never invent features. +Calibration anchors. Apply these literally; they override any generous reading: +- Visual preview: 0 unless the README embeds an actual image, GIF, or video. Icons and links do not count. +- Secrets marked sensitive: README examples that inline literal or placeholder keys (like api_key = "xxxx-xxxxx-xxxx") count as inline secrets, capping this criterion at half even when module inputs are marked sensitive. +- AI governance covers AI Gateway AND Agent Firewall. Full credit requires both documented; exactly one documented earns half. Support that a README says was dropped or removed counts as absent. +- Session continuity: 0 unless the README explicitly documents resuming sessions or running in a persistent session manager. +- Egress transparency: without a dedicated network/offline/air-gapped README section, at most half, no matter how many endpoints appear across examples. +- Mirrorable artifact source and Bring-your-own binary are distinct. Offline or skip-install modes belong to Bring-your-own binary ONLY and never earn Mirrorable credit. To give ANY Mirrorable credit you must name, in the notes, the exact module variable whose value replaces the tool's download URL in the install path. Install-location variables (like install_prefix), version pins, and cache/offline toggles are not download URL overrides. If you cannot name such a variable, score 0. Never use "could be mirrored", "effectively", or "spirit of the criterion" reasoning. +- Egress transparency full credit requires the dedicated section to enumerate the actual endpoints or domains contacted. A section that describes offline behavior without listing endpoints earns half. +- Runs without sudo: inspect the actual scripts. Full only if they never invoke sudo or degrade gracefully when it is absent. Root required for core functionality scores 0. Sudo used only for optional features with a working non-root fallback earns half. +- Restricted-Environment N/A: the download-related criteria (Mirrorable artifact source, Bring-your-own binary, Egress transparency) go N/A when the module downloads or installs nothing of its own (for example, it only invokes tools already in the image or calls the Coder API). A module whose only external interaction is operating on user-provided URLs (like cloning a repo the caller specifies) downloads nothing of its own. Runs without sudo applies whenever the module executes any script, and goes N/A only for modules with no scripts at all. Score 0 only when a concern exists and the module lacks the capability. +- Runs without sudo is an exception to the documentation requirement: scripts that verifiably never invoke sudo earn full credit from the code alone, no README mention needed. +- Half credit is exactly half the criterion's max points, not an arbitrary fraction. +- A perfect theme score should be rare. If you scored every criterion in a theme full, re-check each against its disqualifiers before finalizing. + Output ONLY the scorecard markdown in EXACTLY this structure (this example shows an Agent module; use IDE Integration or omit the track section for Utility): -| Presentation & Onboarding | Agent Integration | Credential Hygiene | Restricted-Network Readiness | Engineering Quality | Overall | +| Presentation & Onboarding | Agent Integration | Credential Hygiene | Restricted-Environment Readiness | Engineering Quality | Overall | |---:|---:|---:|---:|---:|---:| | **X / 25** | **X / 25** | **X / 20** | **X / 20 or N/A** | **X / 10** | **X / 100** | @@ -304,7 +329,7 @@ function prReportSection( ["Presentation & Onboarding", baseline.presentation, summary.presentation], ["Integration", baseline.integration, summary.integration], ["Credential Hygiene", baseline.credential, summary.credential], - ["Restricted-Network", baseline.network, summary.network], + ["Restricted-Environment", baseline.network, summary.network], ["Engineering Quality", baseline.engineering, summary.engineering], ["**Overall**", `**${baseline.overall}**`, `**${summary.overall}**`], ]; diff --git a/.github/scorecard/scorecard-index.ts b/.github/scorecard/scorecard-index.ts index 768293c02..7d12a2f77 100644 --- a/.github/scorecard/scorecard-index.ts +++ b/.github/scorecard/scorecard-index.ts @@ -113,7 +113,7 @@ function buildBody(results: Record): string { ${blurb} -| Module | Presentation & Onboarding |${integrationHeader} Credential Hygiene | Restricted-Network | Engineering Quality | Overall | +| Module | Presentation & Onboarding |${integrationHeader} Credential Hygiene | Restricted-Environment | Engineering Quality | Overall | |---|---:|${integrationSep}---:|---:|---:|---:| ${body} `; @@ -144,14 +144,16 @@ ${body} .join("\n"); const now = new Date().toISOString().slice(0, 10); - return `Every module in the \`coder\` namespace, scored against [SCORECARD.md](https://github.com/coder/registry/blob/main/.github/scorecard/SCORECARD.md). Each module links to its dedicated discussion, where you can share thoughts, questions, and feedback. + return `Every module in the \`coder\` namespace gets a scorecard: a 0-100 score for how well it covers the things we care about in a module, like clear configuration examples, secret handling, air-gapped installs, tests, and Coder platform integration. Each module below links to a dedicated discussion where its scorecard lives and where you can leave thoughts, questions, and feedback. -**Why a scorecard?** It gives us a shared definition of quality (and a goalpost) for modules: what we care about for each one, in one place, for any contributor. It is not a strict gate today, though it may become one, and the criteria will evolve over time. It also helps catch regressions, for example a feature or documented capability being removed, since scores are re-evaluated whenever a module changes. +We built this to have a shared definition of quality for modules. Anyone contributing can see what a good module looks like in one place instead of rediscovering it in review. It is not a gate today (it may become one), and the criteria will change over time. It also catches regressions: modules are re-scored when they change, so a removed feature or deleted doc shows up as a dropped score. -**Looking for a way to contribute?** Low scores are contribution opportunities. Pick a module, open its discussion to see exactly which criteria it misses (visual previews, air-gapped install docs, tests, session persistence, and so on), and open a PR against [\`registry/coder/modules/\`](https://github.com/coder/registry/tree/main/registry/coder/modules). +Scores come from Claude reading each module's code, tests, and README against [SCORECARD.md](https://github.com/coder/registry/blob/main/.github/scorecard/SCORECARD.md), with calibration rules to keep it strict and totals computed by script. A GitHub Action re-scores a module when it changes on \`main\`, re-scores everything weekly or when the criteria change, and comments on PRs that would move a module's score. The scores are advisory. If one looks wrong, say so in the module's discussion. + +**Want to contribute?** Low scores are a ready-made backlog. Pick a module, open its discussion to see exactly which criteria it misses (a visual preview, air-gapped install docs, tests, session persistence), and open a PR against [\`registry/coder/modules/\`](https://github.com/coder/registry/tree/main/registry/coder/modules). ${sections} -**Notes**: N/A means the theme does not apply by construction and is excluded from the denominator. +**Notes**: N/A means the theme does not apply to that module (for example, a module that downloads nothing has no install URL to mirror) and is excluded from its total. --- Updated ${now}. Scores refresh when modules change.