While reviewing PR #503, we noticed that the ThreatModelAssessor remediation includes the step "Point SECURITY.md at the threat model for scope guidance", but no assessor actually checks whether this is done. The remediation advises users to do something that won't affect their score.
This may be a broader pattern. We should audit all assessor remediation instructions and fix any misalignments:
- Each remediation step should map to something the assessor actually scores. Steps that don't affect the score are misleading, since users following remediation expect their score to improve.
- No scored criteria should lack corresponding remediation guidance. If an assessor checks for something, the remediation should tell users how to satisfy it.
- Remediation steps should be actionable and clear. Jargon or vague instructions (like "Point SECURITY.md at the threat model for scope guidance") should be rewritten as concrete actions.
For each misalignment found, use judgment on the right fix:
- If the remediation step reflects something genuinely valuable to assess, add scoring logic to the assessor so the step is no longer unscored.
- If the remediation step is aspirational or out of scope for the assessor, remove or scale back the instruction so users aren't misled.
- If a scored criterion has no remediation guidance, add a remediation step explaining how to satisfy it.
Origin: Found during review of #503, where ThreatModelAssessor._create_remediation() includes an unscored step.
Suggested approach: Iterate through all BaseAssessor subclasses, compare each _create_remediation() (or equivalent) against the assess() scoring logic, and fix misalignments. Update docs/attributes.md for any scoring or remediation changes. Add tests for any new scoring logic.
While reviewing PR #503, we noticed that the
ThreatModelAssessorremediation includes the step "Point SECURITY.md at the threat model for scope guidance", but no assessor actually checks whether this is done. The remediation advises users to do something that won't affect their score.This may be a broader pattern. We should audit all assessor remediation instructions and fix any misalignments:
For each misalignment found, use judgment on the right fix:
Origin: Found during review of #503, where
ThreatModelAssessor._create_remediation()includes an unscored step.Suggested approach: Iterate through all
BaseAssessorsubclasses, compare each_create_remediation()(or equivalent) against theassess()scoring logic, and fix misalignments. Updatedocs/attributes.mdfor any scoring or remediation changes. Add tests for any new scoring logic.