Skip to content

Allow skills to override or configure their computed security score #49

@initializ-mk

Description

@initializ-mk

Summary

The security risk score for a skill is computed entirely from hardcoded package-level rules in forge-skills/analyzer/scoring.go. Skill authors and operators have no supported way to override the score, configure the scoring tables, or load a custom security policy when running forge skills audit.

This is limiting for custom skills that legitimately need a high-risk binary, a non-builtin egress domain, or a script, but are reviewed/approved at the operator level and should not be flagged repeatedly by the default policy.

Current behavior

AnalyzeSkillEntry (forge-skills/analyzer/scoring.go:70) sums RiskFactor points from four categories using package-level constants:

Category Rule Source
egress +2 per trusted domain, +10 per unknown, +15 if >5 domains scoring.go:116-152
binary +15 per high-risk binary, +3 per standard scoring.go:154-172
env +10 per sensitive env name, +5 otherwise scoring.go:174-197
script +20 if skill has a script scoring.go:199-205

The trustedDomains, highRiskBinaries, and sensitiveEnvPatterns lists (scoring.go:11-47) are unexported package vars — not reachable from SKILL.md, forge.yaml, or any CLI flag.

Gaps in the existing policy plumbing

The SecurityPolicy struct (forge-skills/analyzer/types.go:63-71) already has fields that look like they should influence scoring, but the wiring is incomplete:

  • SecurityPolicy.TrustedDomains exists, but scoreEgress is always invoked with extraTrusted = nil (scoring.go:53, scoring.go:100). The policy's trusted list never reaches the scorer.
  • SecurityPolicy.MaxRiskScore is checked only inside CheckPolicy to emit a max_risk_score violation (policy.go:99-109); it doesn't influence the score value itself.
  • The forge skills audit command hardcodes analyzer.DefaultPolicy() (forge-cli/cmd/skills.go:377) with no flag to load an alternative policy.

So even an operator who builds a SecurityPolicy today cannot make scoring honor it through the CLI.

Why a static list in analyzer/scoring.go is not the fix

Adding a new domain to trustedDomains, a new env pattern to sensitiveEnvPatterns, or a new binary to highRiskBinaries requires a code change to forge-skills/analyzer/scoring.go every time a custom skill needs different scoring treatment. This does not scale and is the same anti-pattern called out in #48 for runner.go's hardcoded knownKeys.

The solution must be dynamic: a custom skill author or operator must be able to influence scoring without editing any Go file in the forge repo.

Proposed options (any one or combination)

  1. Honor SecurityPolicy.TrustedDomains in scoring. Thread the policy through AnalyzeSkillEntry / AnalyzeSkillDescriptor so scoreEgress receives policy.TrustedDomains as extraTrusted. This closes the existing gap without any new schema.
  2. --policy <path> flag on forge skills audit. Load a SecurityPolicy YAML, replacing DefaultPolicy() at forge-cli/cmd/skills.go:377. Combined with (1), this lets operators expand trusted domains, binary lists, and sensitive-env patterns per-project.
  3. forge.security block in SKILL.md frontmatter. Let a skill declare its own scoring inputs:
    • trusted_domains: [<extra domains specific to this skill>]
    • acknowledged_high_risk_bins: [<list>] with a justification field — does not silence the factor, but caps its points or marks it as accepted.
    • Optionally a score_override with a max_value and required justification so the override is auditable.
  4. forge.yaml project-level policy block. A security_policy: section in forge.yaml that loads the same SecurityPolicy shape, applied to all skills in the project. Avoids per-invocation flags.

Approach (1)+(2) is the minimum to fix the existing wiring gap. (3) and (4) are user-facing enhancements that build on top.

Affected files (pointers, not a fix list)

  • forge-skills/analyzer/scoring.go:11-47 — hardcoded tables
  • forge-skills/analyzer/scoring.go:50-114AnalyzeSkillDescriptor / AnalyzeSkillEntry (no policy parameter today)
  • forge-skills/analyzer/scoring.go:116scoreEgress accepts extraTrusted but is always called with nil
  • forge-skills/analyzer/types.go:63-71SecurityPolicy (existing fields not threaded into scoring)
  • forge-skills/analyzer/policy.go:11-24DefaultPolicy
  • forge-cli/cmd/skills.go:377 — hardcoded analyzer.DefaultPolicy() in audit
  • forge-cli/cmd/skills.go:91-93 — audit command flags (no --policy)

Acceptance criteria

  • A custom skill can have its egress domain, env var, or binary scored as non-risky without editing any file under forge-skills/analyzer/.
  • forge skills audit accepts (or reads) a configurable policy that affects both scoring and policy checks, not only the latter.
  • Any score override path is auditable — i.e. it leaves a trace in the audit report (factor entry, justification, or similar) so operators can see the override was applied rather than the score being silently lower.
  • Default behavior is unchanged for skills that declare no overrides and run under no custom policy.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions