Problem
Cockpit's Red team scans readiness item uses an obsolete and unreliable signal:
- it searches
.agentops/bundles/, although the named-bundle layout is legacy and no longer part of the product;
- it recommends
safe_agent_baseline.yaml, which is not shipped;
- it ignores the current
redteam: configuration in agentops.yaml;
- it ignores normalized results written by
agentops redteam run to .agentops/redteam/latest.json;
- it reports ready from configuration-file presence rather than a completed scan and its outcome;
- it recommends running both AgentOps and Foundry scans without explaining when one source is sufficient.
Adversarial testing is valuable for every production-bound GenAI agent, but readiness must be based on a real, current scan against the relevant target—not on a legacy YAML file.
Expected behavior
Replace the legacy card with a verified red-team release gate:
- Before
agentops init, do not show red-team readiness.
- For an initialized workspace with no scan evidence, show a readiness warning with the current
agentops redteam run workflow and Foundry as supported alternatives.
- Use the normalized
.agentops/redteam/latest.json result when agentops redteam run was used.
- Accept a configured
redteam_path artifact from a native Foundry scan when it can be normalized and validated.
- Do not require users to run both AgentOps and Foundry scans.
- Report ready only when the scan completed, covers the configured target and required risk categories, and stays within the configured attack-success threshold.
- Treat a threshold failure as blocking release evidence and a critical Doctor finding.
- Treat missing, malformed, stale, or target-mismatched evidence as not ready with actionable remediation.
- Record enough metadata in normalized results to validate freshness and target alignment, including generation time and a stable fingerprint of the available target/configuration inputs.
- Require a new scan after relevant changes to the model, prompt, tools, permissions, or red-team policy.
The depth and frequency of red teaming may remain risk-based, but a production release must not be described as red-team ready without verified evidence.
Acceptance criteria
- No Cockpit, Doctor, documentation, or test references the legacy
.agentops/bundles/ red-team detector or safe_agent_baseline.yaml.
- An uninitialized workspace does not show the red-team readiness item.
- An initialized workspace without scan evidence reports a clear readiness warning.
- A configured-but-unexecuted
redteam: block is not treated as ready.
- A current passing AgentOps scan is shown as ready with target, timestamp, covered risk categories, attack success rate, and threshold.
- A supported Foundry scan artifact can satisfy the same readiness contract without a duplicate AgentOps scan.
- Failed thresholds become blocking/critical evidence using the existing AgentOps exit-code and severity contracts.
- Malformed, stale, or target-mismatched results do not satisfy readiness.
- Normalized results contain the metadata needed for freshness and target-alignment checks.
- Unit tests cover missing, passing, failing, stale, mismatched, and Foundry-imported evidence.
Problem
Cockpit's Red team scans readiness item uses an obsolete and unreliable signal:
.agentops/bundles/, although the named-bundle layout is legacy and no longer part of the product;safe_agent_baseline.yaml, which is not shipped;redteam:configuration inagentops.yaml;agentops redteam runto.agentops/redteam/latest.json;Adversarial testing is valuable for every production-bound GenAI agent, but readiness must be based on a real, current scan against the relevant target—not on a legacy YAML file.
Expected behavior
Replace the legacy card with a verified red-team release gate:
agentops init, do not show red-team readiness.agentops redteam runworkflow and Foundry as supported alternatives..agentops/redteam/latest.jsonresult whenagentops redteam runwas used.redteam_pathartifact from a native Foundry scan when it can be normalized and validated.The depth and frequency of red teaming may remain risk-based, but a production release must not be described as red-team ready without verified evidence.
Acceptance criteria
.agentops/bundles/red-team detector orsafe_agent_baseline.yaml.redteam:block is not treated as ready.