[Backlog]: Tool Invocation and Function Calling Exploitation in Agentic Systems #33
Description
Checklist
- Backlog entry requires creating new sandboxes.
- Backlog entry requires creating new exploitation code and/or tutorials.
CVE List
No response
Description
Summary
Exploitation of tool invocation and function calling mechanisms in agentic GenAI systems to induce unintended actions, command execution, or privilege escalation.
Reference: https://owasp.org/www-project-top-10-for-large-language-model-applications/
GenAI Red Teaming Manual Reference
4.3 Agentic Systems & Tool Use (or closest applicable section)
Sandbox
Reuse sandbox sandboxes/agentic_local_n8n_v1.65.0 (#22) and/or other agent-based sandboxes.
Exploitation
Inject crafted prompts that manipulate the model into invoking unintended tools or functions.
Examples include:
- Prompt injection to force execution of restricted tools (e.g., shell commands, API calls)
- Crafting inputs that result in command injection via tool interfaces
- Chaining multi-step agent actions to escalate privileges or bypass safeguards
- Abusing structured outputs (e.g., JSON function calls) to trigger unsafe operations