Skip to content

Replace legacy red-team readiness detection with verified scan evidence #454

Description

Problem

Cockpit's Red team scans readiness item uses an obsolete and unreliable signal:

  • it searches .agentops/bundles/, although the named-bundle layout is legacy and no longer part of the product;
  • it recommends safe_agent_baseline.yaml, which is not shipped;
  • it ignores the current redteam: configuration in agentops.yaml;
  • it ignores normalized results written by agentops redteam run to .agentops/redteam/latest.json;
  • it reports ready from configuration-file presence rather than a completed scan and its outcome;
  • it recommends running both AgentOps and Foundry scans without explaining when one source is sufficient.

Adversarial testing is valuable for every production-bound GenAI agent, but readiness must be based on a real, current scan against the relevant target—not on a legacy YAML file.

Expected behavior

Replace the legacy card with a verified red-team release gate:

  • Before agentops init, do not show red-team readiness.
  • For an initialized workspace with no scan evidence, show a readiness warning with the current agentops redteam run workflow and Foundry as supported alternatives.
  • Use the normalized .agentops/redteam/latest.json result when agentops redteam run was used.
  • Accept a configured redteam_path artifact from a native Foundry scan when it can be normalized and validated.
  • Do not require users to run both AgentOps and Foundry scans.
  • Report ready only when the scan completed, covers the configured target and required risk categories, and stays within the configured attack-success threshold.
  • Treat a threshold failure as blocking release evidence and a critical Doctor finding.
  • Treat missing, malformed, stale, or target-mismatched evidence as not ready with actionable remediation.
  • Record enough metadata in normalized results to validate freshness and target alignment, including generation time and a stable fingerprint of the available target/configuration inputs.
  • Require a new scan after relevant changes to the model, prompt, tools, permissions, or red-team policy.

The depth and frequency of red teaming may remain risk-based, but a production release must not be described as red-team ready without verified evidence.

Acceptance criteria

  • No Cockpit, Doctor, documentation, or test references the legacy .agentops/bundles/ red-team detector or safe_agent_baseline.yaml.
  • An uninitialized workspace does not show the red-team readiness item.
  • An initialized workspace without scan evidence reports a clear readiness warning.
  • A configured-but-unexecuted redteam: block is not treated as ready.
  • A current passing AgentOps scan is shown as ready with target, timestamp, covered risk categories, attack success rate, and threshold.
  • A supported Foundry scan artifact can satisfy the same readiness contract without a duplicate AgentOps scan.
  • Failed thresholds become blocking/critical evidence using the existing AgentOps exit-code and severity contracts.
  • Malformed, stale, or target-mismatched results do not satisfy readiness.
  • Normalized results contain the metadata needed for freshness and target-alignment checks.
  • Unit tests cover missing, passing, failing, stale, mismatched, and Foundry-imported evidence.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions