feat(rules): add AI600-AI900 security rules for agents, RAG, API keys, and output handling

[mmustafasenoglu]

Summary

Adds 18 new AI/LLM security rules covering understimated attack surfaces: unsafe agent behavior, RAG pipeline security, hardcoded API keys, and unsafe output handling.

Motivation

Closes #91

The existing ruleset covers prompt injection and model deserialization well, but modern LLM applications have expanded attack surfaces that weren't being caught. This adds coverage for:

Changes

AI600 - Unsafe Agent Behavior & Tool Poisoning

• AI601: Unrestricted web browsing in agents (SSRF risk)
• AI602: Subprocess execution via agent tools (arbitrary command execution)
• AI603: Unrestricted file write capability in agents
• AI604: Indirect prompt injection via unsanitized tool output
• AI605: Verbose agent logging exposing sensitive data
• AI606: Disabled parsing error handling leaking internal state

AI700 - RAG Security

• AI701: Unvalidated document ingestion (embedding poisoning)
• AI702: Low similarity threshold retrieving adversarial content
• AI703: Context injection without size limits (DoS)
• AI704: Embedding models from untrusted sources

AI800 - API Key & Credential Management

• AI801: Hardcoded OpenAI API keys
• AI802: Hardcoded Anthropic API keys
• AI803: API keys in URL query parameters
• AI804: Hardcoded Cohere API keys

AI900 - Output Handling & DoS

• AI901: Unsafe YAML parsing of LLM output (RCE via yaml.load)
• AI902: JSON parsing without size limits (memory exhaustion)
• AI903: exec/eval of LLM-generated code
• AI904: Unsanitized LLM output rendered as HTML (XSS)

Also adds 4 new taint sources/sinks (AITS11-12, AISK11-12) for RAG vector store retrieval and agent web tool flows.

Testing

• All new rules have TOML metadata validation
• Pattern matching tests for each rule category
• Exclude pattern tests for safe alternatives (e.g., yaml.safe_load)
• TOML file validates successfully (44 total rules, 12 taint sources, 12 sinks)

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[desimetallica]

Hello @mmustafasenoglu thanks for adding the AI600-AI900 rules. The security coverage is valuable, but I recommend requesting changes before merging because several rules currently produce false positives or do not match their stated behavior.

Some comments:

• AI601: the regex misses common multiline/separately-defined agent tools, flags unrelated code, and its test example does not match the regex. AITS12 also taints every requests.get() response, while AISK11 duplicates the existing SSRF sink.
• AI603: open\s*\(.*['"]w['"]|['"]a['"] has an ungrouped alternation and matches unrelated code such as mode = "a". It also flags ordinary file writes without establishing agent context.
• AI604, AI903, AI904: broad regex alternatives create significant false positives. In particular, AI903 and AI904 can match any line containing response, output, or completion.
• AI702: the rule claims to detect a low similarity threshold, but the pattern checks k=, which is the number of retrieved documents, not a similarity threshold.
• AI704, AI701, AI602, AI605, AI703, AI902: these rules could flag common legitimate APIs without proving that they are used in an agent/RAG/security-sensitive context.
• AI704 and AITS11: AITS11 contains a malformed function path and will not reliably match normal instance calls such as vectorstore.similarity_search(...).

The tests currently focus mostly on positive regex matches. Please add negative tests for ordinary HTTP clients, subprocess usage, JSON parsing, trusted embedding models, normal file writes, sanitized HTML, and multiline code. Taint-flow tests should also verify that trusted constants do not become tainted and that duplicate AI501/AI601 findings are avoided.

I suggest narrowing or removing the broad regex rules and using AST/taint analysis where the rule depends on data flow or agent context. After these corrections, the proposed rule categories should be much more useful with less noise.

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[mmustafasenoglu]

Thanks for the detailed review, @desimetallica — you're right on every point. I've reworked the rules and addressed all of it in 2a73c99.

Removed (couldn't reliably prove agent context with line-based regex, or duplicated existing coverage):

• AI601 + AITS12 + AISK11: the regex missed multiline tool definitions and the test example never matched. Agent-side SSRF is already covered by AI501/AISK08 taint analysis, and AISK11 was a duplicate of AISK08. AITS12 tainting every requests.get() response was too broad, so it's gone too.
• AI603: the ungrouped alternation did match mode = "a". Agent-side filesystem access is already covered by AI502/AISK09.
• AI604: unsanitized tool-output → prompt is a data-flow issue; the equivalent flow is already covered by the AITS04 → AISK01 (AI101) taint path.

Narrowed:

• AI602: now only fires on subprocess.(run|call|Popen|check_output)(...) with shell=True, so ordinary subprocess usage is ignored.
• AI702: you're right that k= is the retrieval count, not a similarity threshold. The pattern now looks for an explicit score_threshold=0.x instead.
• AI902 / AI903 / AI904: the .*response|.*output|.*completion alternatives matched any line containing those words — that was bad. Patterns are now scoped to actual LLM-output variables (e.g. eval(llm_output), json.loads(response)), and AI904 got a sanitize|bleach exclude for sanitized HTML.
• AITS11: fixed the malformed path (stray space) and switched to .similarity_search so normal instance calls like vectorstore.similarity_search(...) match.

Lowered confidence / clarified: AI605, AI701, AI703, AI704 now carry confidence = "Low" with descriptions that no longer overclaim agent/RAG context.

New tests (68 in total now):

• Negative: plain HTTP clients, subprocess without shell=True, plain json.loads/eval, trusted embedding models, sanitized innerHTML, top-k without threshold, verbose=False, plain file reads, multiline requests.get.
• Taint-flow: trusted constant URLs stay untainted; tainted URL → requests.get fires AI501 but not AI601 (no duplicate findings); AITS11 retrieval → prompt flow fires AI101.

The broader rules that depend on data flow or agent context are now taint-driven rather than regex-driven, so the remaining regexes should produce much less noise.