AI Red-Teaming & Security-Testing Tooling for LLM Apps (2024-2026)
Parent: Global AI Hub Research Corpus · researched 2026-05-31· 1 source · 0 concepts
AI red-teaming is the offensive testing discipline for LLM and generative-AI applications: systematically generating adversarial inputs to find where a model or app fails in ways we don't want (jailbr
Overview
- AI red-teaming is the offensive testing discipline for LLM and generative-AI applications: systematically generating adversarial inputs to find where a model or app fails in ways we don't want (jailbreaks, prompt injection, data/system-prompt exfiltration, harmful-content generation, insecure tool use), then feeding those findings back into fixes and regression tests. It is distinct from runtime guardrails/defense (input/output filters, dual-LLM/CaMeL), which are the defensive counterpart and are covered by the existing agent-reliability-guardrails reference - this report focuses on the testing tooling and methodology and cross-references the defensive layer where relevant. [source]
- By 2025-2026 the field has matured from ad-hoc manual probing into a tooled discipline with: (1) open-source scanners and frameworks (Garak, PyRIT, promptfoo, Giskard, Purple Llama); (2) standardized benchmarks (HarmBench, JailbreakBench, AdvBench, AgentDojo); (3) automated attack-generation algorithms (GCG, PAIR, TAP, Crescendo, many-shot); (4) commercial continuous red-teaming platforms (Lakera/Cisco, Mindgard, HiddenLayer, Robust Intelligence/Cisco AI Defense); and (5) governance/process frameworks (OWASP LLM Top 10 2025, OWASP GenAI Red Teaming Guide, MITRE ATLAS, NIST AI RMF). The dominant consensus lesson is that automation augments but does not replace human red-teamers, and that red-teaming must be continuous, not one-off (Microsoft Security Blog; Mend.io). [source]
1. AI Red-Teaming as a Discipline (offensive testing) — *Confidence: High*
- Definition & posture. Red-teaming starts by defining a threat model: a description of the AI system, the relevant vulnerabilities, and the contexts in which they arise, including human interactions (CSET Georgetown). It is offensive testing, distinct from blue-team runtime defenses. [source]
- Manual vs automated. Automated tools (PyRIT, Garak, commercial platforms) excel at systematic coverage and regression testing but cannot match human creativity in discovering novel attacks; manual expert testing remains essential (Mend.io; Vectra). Microsoft's own framing: PyRIT "is not a replacement for manual red teaming... it augments an AI red teamer's existing domain expertise and automates the tedious tasks" (Microsoft Security Blog, 2024). [source]
- Continuous red-teaming. The 2025-2026 shift is toward continuous adversarial testing run regularly in staging or production monitoring, rather than periodic one-shot engagements (Ajith Prabhakar playbook; Mindgard). Regression testing after fixes is treated as mandatory, not optional (Mend.io). [source]
- Microsoft's lessons from 100 products. Eight key lessons include: start from downstream impact rather than attack mechanics; prompt engineering often outperforms gradient-based attacks in practice; and red-teaming cannot be replaced by safety benchmarks. Their AIRT ontology models a finding as: an Actor conducts an Attack leveraging TTPs to exploit a Weakness in a System, creating an Impact (Lessons From Red Teaming 100 Generative AI Products, arXiv 2501.07238; Microsoft Security Blog). [source]
2. Tooling Landscape: Scanners vs Frameworks — *Confidence: High*
- The landscape splits into scanners (run a fixed/curated battery of probes, give a vulnerability report) and frameworks (composable building blocks you script your own attacks with). The major open-source tools: [source]
Garak (NVIDIA) — *the leading open-source LLM vulnerability scanner*
- Name = "Generative AI Red-teaming & Assessment Kit." Developed Spring 2023 by Prof. Leon Derczynski; homed at NVIDIA as an open-source project with long-term support since Nov 2024 (Wikipedia); GitHub). [source]
- Five-component plugin architecture: Probes (generate adversarial interactions), Detectors (identify failure modes in responses), Generators (interface to model platforms - OpenAI, HuggingFace, Bedrock, etc.), Harnesses (orchestrate the workflow; default probewise), and Evaluators (produce findings reports) (Garak README). [source]
- 20+ probe modules covering prompt injection (encoding-based, structural), DAN-family jailbreaks, encoding/Base64 guardrail bypass, data leakage, package/dependency hallucination, toxicity, malware generation, XSS via LLM output, and glitch tokens. Notable modules: encoding, dan, promptinject, malwaregen, realtoxicityprompts (Garak README; Databricks blog). [source]
- CLI-driven: garak --target_type openai --target_name <model> --probes encoding. License Apache 2.0 (Garak README). [source]
Microsoft PyRIT — *automation framework, "Metasploit for LLMs"*
- PyRIT = Python Risk Identification Toolkit. Released Feb 2024 (Microsoft Security Blog; The Hacker News). [source]
- Four core components: Targets (the AI under test), Converters (transform/obfuscate prompts before sending), Scorers (evaluate whether responses are harmful), Orchestrators (manage multi-turn conversation flow). It is adaptive - it changes tactics based on the system's response and generates the next input until the goal is reached (Medium deep dive; ToxSec). [source]
- In one Copilot exercise, Microsoft generated several thousand malicious prompts and scored outputs "in hours instead of weeks." Now integrated into Azure AI Foundry's AI Red Teaming Agent (Microsoft Security Blog; Microsoft Learn). [source]
promptfoo — *eval + red-team, CI-native*
- CLI/library for evaluating and red-teaming LLM apps, runs 100% locally; used by OpenAI and Anthropic and 156 of the Fortune 500 (GitHub; Promptfoo docs). [source]
- Built-in vulnerability scanner covering 50+ vulnerability types and OWASP LLM Top 10, with presets aligned to NIST AI RMF, OWASP LLM Top 10, and MITRE ATLAS (Promptfoo MITRE ATLAS; Promptfoo config). [source]
- CI/CD integration: can automatically fail builds if AI components regress or become vulnerable - the canonical "red-team-in-CI" pattern (Promptfoo CI/CD). [source]
Giskard — *open-source scan for security + quality*
- Apache-2.0 Python library that auto-detects security vulns (prompt injection, data leakage) and quality issues (hallucination, bias, sycophancy) in LLMs, RAG systems, and classic ML (Giskard docs; Giskard site). [source]
- Uses a mixture of predefined examples, heuristics, and LLM-assisted detectors (GPT-4-based); notable for sycophancy testing to surface hallucination without ground truth (Giskard docs; HuggingFace blog). [source]
Meta Purple Llama — *guardrail models + the offensive benchmark*
- A toolset spanning Llama Guard (input/output content moderation classifier), Prompt Guard (mDeBERTa-based classifier for jailbreaks + indirect injection), and CyberSecEval (GitHub; Meta Prompt Guard overview). [source]
- For the testing angle, CyberSecEval is the key piece: a benchmark of an LLM's propensity to (a) generate insecure code and (b) comply with cyberattack assistance. CyberSecEval 3 added visual prompt-injection tests, spear-phishing capability tests, and autonomous offensive cyber-operations tests (CyberSecEval paper, arXiv 2312.04724; Meta AI research). (Llama Guard / Prompt Guard themselves are defensive - cross-reference agent-reliability-guardrails.) [source]
3. Attack Taxonomy — *Confidence: High*
- Direct vs indirect prompt injection. Direct injection (often conflated with jailbreaking) is when a user's input overrides/reveals the system prompt. Indirect injection smuggles malicious instructions via untrusted content the model ingests (RAG docs, web pages, files, tool outputs) - here the attacker is not the user but a third party poisoning the data channel (OWASP LLM01:2025; Greshake et al., arXiv 2302.12173 - the foundational indirect-injection paper; CrowdStrike). [source]
- Jailbreak families. Documented families include DAN ("Do Anything Now") roleplay/dual-persona, generic roleplay/persona-escalation and "admin" status, payload splitting, obfuscation/encoding (Base64, multilingual injection, hiding in code comments), and multi-turn gradual escalation (ToxSec DAN; OnSecurity; Astra guide). [source]
- Many-shot jailbreaking (Anthropic, 2024). Prompt the model with hundreds-to-thousands of in-context examples of undesirable behavior, then ask the harmful question; effectiveness follows a power law with shot count and is newly feasible due to long context windows (Anthropic Many-Shot paper PDF; Prompt Security explainer). [source]
- Crescendo (multi-turn). A seemingly benign multi-turn attack that exploits the model's tendency to follow patterns and focus on its own recent output; reaches up to 100% ASR on many tasks and beats prior SOTA by 29-71% on AdvBench subsets (Crescendo paper, arXiv 2404.01833; Crescendo site). [source]
- Data / system-prompt exfiltration. A successful injection in a tool-enabled app can exfiltrate data directly via tools (e.g., writing to a public GitHub repo, DNS exfil, markdown-image rendering) - this is LLM02 Sensitive Information Disclosure and LLM07 System Prompt Leakage in OWASP 2025 (CrowdStrike indirect-injection blog; OWASP LLM01:2025). [source]
- The lethal trifecta (cross-ref to guardrails). Simon Willison's June 2025 framing: an agent is structurally insecure when it combines (1) access to private data, (2) exposure to untrusted content, and (3) the ability to communicate externally - a single poisoned input can then exfiltrate data with no traditional code vuln. As a testing target this maps directly to indirect-injection + tool-poisoning test cases; the defensive mitigation is covered by agent-reliability-guardrails (Simon Willison; Promptfoo lethal-trifecta testing; HiddenLayer). [source]
4. Automated Attack Generation — *Confidence: High*
- GCG (Greedy Coordinate Gradient). Token-level, white-box - optimizes an adversarial suffix via greedy coordinate gradient descent to force affirmative ("Sure, here is…") outputs. High success but orders of magnitude more queries than search-based methods; suffixes can transfer across models (Tree of Attacks paper comparison, arXiv 2312.02119; EmergentMind GCG+PAIR). [source]
- PAIR (Prompt Automatic Iterative Refinement). Uses an attacker LLM steered by meta-prompts to iteratively refine semantically coherent adversarial prompts - black-box, query-efficient (TAP paper). [source]
- TAP (Tree of Attacks with Pruning). Black-box; an attacker LLM iteratively refines a tree of candidate prompts with pruning. Against GPT-4o it finds jailbreaks for 16% more prompts than PAIR with 60% fewer queries. This is the algorithm Robust Intelligence/Cisco productized for algorithmic red-teaming (TAP paper, arXiv 2312.02119; Cisco/Robust Intelligence). [source]
- Red-teamer LLMs & fuzzing. The common modern pattern is an attacker/red-teamer LLM that generates, mutates, and scores attacks in a loop - the core of PyRIT orchestrators, TAP, PAIR, and Crescendo automation ("Crescendomation"). Hybrid GCG+PAIR combines gradient suffixes with semantic refinement for up to +33pp ASR over single-strategy (EmergentMind; Crescendo paper). [source]
5. Benchmarks & Datasets — *Confidence: High*
- AdvBench (Zou et al., 2023) - the original harmful-behaviors/harmful-strings dataset introduced with GCG; now a component of later benchmarks (JailbreakBench paper, arXiv 2404.01318). [source]
- HarmBench (Center for AI Safety, 2024) - standardized evaluation framework for automated red-teaming and robust refusal, broader topic coverage including copyright and multimodal (JailbreakBench paper). [source]
- JailbreakBench (NeurIPS 2024) - open robustness benchmark: 100 harmful + 100 benign behaviors across 10 categories aligned to OpenAI usage policies, a validated jailbreak classifier, an artifact repository, and a public leaderboard for attacks and defenses (JailbreakBench GitHub; paper, arXiv 2404.01318). [source]
- AgentDojo (2024) - the key benchmark for tool-use / agent injection: a dynamic environment with 97 realistic tasks and 629 security test cases across banking, Slack, travel, and workspace domains; metrics are benign utility, utility-under-attack, and attack success rate. Used by US/UK AISI to demonstrate Claude 3.5 Sonnet's injection vulnerability (AgentDojo paper, arXiv 2406.13352; AgentDojo site). [source]
- OWASP LLM Top 10 (2025) as a testing checklist. Functions as a structured 10-category test plan for any production LLM deployment (Mend.io 2025 guide): LLM01 Prompt Injection, LLM02 Sensitive Information Disclosure, LLM03 Supply Chain, LLM04 Data & Model Poisoning, LLM05 Improper Output Handling, LLM06 Excessive Agency, LLM07 System Prompt Leakage, LLM08 Vector & Embedding Weaknesses, LLM09 Misinformation, LLM10 Unbounded Consumption (OWASP LLM01:2025; Oligo). [source]
6. AI Firewalls / Runtime-Defense Products from the Testing Angle — *Confidence: High*
- These are defensive products, but each ships a red-team/testing capability or dataset relevant to offensive testing: [source]
- Lakera - Gandalf & Lakera Red. Gandalf is a gamified prompt-injection challenge (extract a secret password through 7-8 escalating defense levels); it has drawn 1M+ players and 80M+ adversarial prompts that feed Lakera's threat intel. Lakera Red runs automated attack simulations against LLM apps pre-production. Lakera (incl. Guard, Red, Gandalf dataset) was acquired by Cisco in 2025 (Lakera "Who is Gandalf"; Lakera prompt-injection guide). [source]
- Robust Intelligence → Cisco AI Defense. Robust Intelligence (Harvard spinout, founded 2019) built what it calls the first algorithmic red-teaming solution, using TAP to jailbreak LLMs in seconds rather than weeks of human effort. Acquired by Cisco Oct 2024; now powers Cisco AI Defense "AI Validation" (Cisco; Cisco blog). [source]
- Mindgard. Automated AI red-teaming / DAST-AI platform for LLMs, agents, multimodal; Lancaster University spinout (2022), recognized in the OWASP GenAI Security Solutions Landscape; emphasizes continuous red-teaming (Mindgard; Mindgard/S&P Global). [source]
- HiddenLayer & Prompt Guard. HiddenLayer's AISec platform focuses on ML model protection and threat intel; Meta's Prompt Guard classifier is the open-source guardrail you test against (Mend.io top-6 services; Meta Prompt Guard). [source]
7. Process: Scoping, CI, Reporting, and Framework Mapping — *Confidence: High*
- OWASP GenAI Red Teaming Guide (Jan 2025) structures red-teaming into four phases: (1) model evaluation (provenance, data pipelines), (2) implementation testing (guardrails in place), (3) infrastructure/system assessment (deployed exploitable components), and (4) runtime behavior analysis (business processes and multi-component interactions in production) (OWASP GenAI Red Teaming Guide; ResilientCyber walkthrough; CSO Online). [source]
- Three governance frameworks, three lifecycle phases. The standard mapping: OWASP LLM Top 10 for the development/secure-coding phase, MITRE ATLAS for operations/threat-modeling/detection, NIST AI RMF (Govern/Map/Measure/Manage) for governance/compliance - they complement, not compete (Straiker; Medium - Elias Silva). [source]
- MITRE ATLAS scale. As of the Nov 2025 v5.1.0 update: 16 tactics, 84 techniques, 32 mitigations, 42 case studies, with continued additions of agentic-AI techniques into 2026 (Vectra; Repello). OWASP published a first-ever Red Teaming Landscape in 2026 alongside separate LLM/GenAI and agentic-AI landscapes (Straiker). [source]
- Red-team in CI + reporting. promptfoo is the reference implementation: run the red-team suite in CI/CD and fail the build on regression, with reports mapped to OWASP/NIST/ATLAS (Promptfoo CI/CD). Best-practice reporting documents every scenario with goals, prompts, model versions, and observed behaviors as reproducible test cases (Ajith Prabhakar playbook). [source]
8. Anti-Patterns — *Confidence: High*
- One-off red-teaming. Treating red-teaming as a single pre-launch event; models, prompts, and threat landscape drift, so testing must be continuous (Mindgard; Mend.io). [source]
- Testing only direct injection. Ignoring indirect injection / tool-poisoning (RAG docs, web content, tool outputs) - the highest-impact agentic attack surface and the core of the lethal trifecta (Greshake et al.; AgentDojo). [source]
- No regression after fixes. Failing to re-run the attack that found a bug after patching; "regression testing is not optional in AI security" (Mend.io). [source]
- Treating benchmarks as a substitute for red-teaming. Microsoft explicitly warns against the idea that safety benchmarks can replace red-teaming - benchmarks measure known behaviors; red-teaming finds novel failures (Lessons From Red Teaming 100, arXiv 2501.07238). [source]
- Over-relying on gradient attacks. In real products, prompt engineering often outperforms gradient-based attacks (GCG-style) and is far cheaper; teams that fixate on academic white-box attacks miss the easy wins (Microsoft Security Blog). [source]
- Automation-only. Assuming an automated scanner is sufficient; it cannot match human creativity for novel attacks (Vectra; Microsoft, 2024). [source]
9. Suggested Child Sub-Concepts (6-10 future concepts)
- Open-source red-team scanners - Garak, Giskard, promptfoo red-team (scanner-vs-framework, probe/detector architecture, CI integration). [source]
- Adversarial attack-generation algorithms - GCG, PAIR, TAP, fuzzing, red-teamer LLMs (white-box vs black-box, query efficiency, transferability). [source]
- Jailbreak families & prompt-injection taxonomy (offensive) - DAN/roleplay, payload splitting, encoding, many-shot, Crescendo; direct vs indirect; exfiltration patterns. [source]
- Agent / tool-use injection testing - AgentDojo, indirect injection, tool-poisoning, lethal-trifecta test cases (cross-ref guardrails). [source]
- Red-team benchmarks & datasets - AdvBench, HarmBench, JailbreakBench, CyberSecEval; leaderboards and validated classifiers. [source]
- PyRIT & orchestration frameworks - Targets/Converters/Scorers/Orchestrators, adaptive multi-turn automation, Azure AI Foundry Red Teaming Agent. [source]
- Red-team process & governance mapping - OWASP GenAI Red Teaming Guide 4 phases, OWASP LLM Top 10 as test plan, MITRE ATLAS, NIST AI RMF; red-team-in-CI and reporting. [source]
- Commercial continuous red-teaming platforms - Lakera/Gandalf, Cisco AI Defense (Robust Intelligence), Mindgard, HiddenLayer (DAST-AI, algorithmic red-teaming). [source]
Knowledge Gaps
- Quantitative tool comparisons (precision/recall of Garak vs promptfoo vs Giskard detectors on a common suite) are scarce in public sources - vendor claims dominate; treat comparative effectiveness as Low confidence. [source]
- Robust Intelligence post-acquisition specifics beyond the TAP-based algorithmic red-teaming and "AI Validation" branding are thin in public sources. [source]
- Multimodal and voice red-teaming tooling (image/audio injection) is emerging (CyberSecEval 3 visual injection; ATLAS agentic additions) but tooling maturity is unclear - flagged as a frontier, not a settled area. [source]
Sources
- NVIDIA/garak GitHub - the LLM vulnerability scanner repo. [source]
- Garak README (raw) - architecture, probes, CLI, Apache-2.0 license. [source]
- Garak (software) - Wikipedia) - history, ownership, recognition. [source]
- Databricks - Applying Garak - probe categories in practice. [source]
- Microsoft Security Blog - Announcing PyRIT (2024) - PyRIT launch, augments-not-replaces framing. [source]
- The Hacker News - Microsoft releases PyRIT - overview. [source]
- Medium - Automating AI Red Teaming with PyRIT - Targets/Converters/Scorers/Orchestrators. [source]
- Microsoft Learn - AI Red Teaming Agent (Foundry) - PyRIT in Azure AI Foundry. [source]
- Promptfoo - LLM red teaming docs - scanner scope, 50+ vuln types. [source]
- promptfoo/promptfoo GitHub - used by OpenAI/Anthropic, CI/CD. [source]
- Promptfoo - CI/CD integration - fail builds on regression. [source]
- Promptfoo - MITRE ATLAS preset - framework mapping. [source]
- Promptfoo - testing the lethal trifecta - trifecta as test cases. [source]
- Giskard - LLM vulnerabilities docs - scan categories. [source]
- Giskard - LLM detectors reference - sycophancy/hallucination detectors. [source]
- meta-llama/PurpleLlama GitHub - Llama Guard, Prompt Guard, CyberSecEval. [source]
- Meta - Prompt Guard overview - jailbreak + indirect-injection classifier. [source]
- CyberSecEval paper (arXiv 2312.04724) - secure-coding + cyberattack-compliance benchmark. [source]
- Tree of Attacks paper (arXiv 2312.02119) - TAP vs PAIR vs GCG comparison. [source]
- Crescendo paper (arXiv 2404.01833) - multi-turn jailbreak. [source]
- Anthropic - Many-Shot Jailbreaking (PDF) - power-law many-shot attack. [source]
- Greshake et al. - Indirect Prompt Injection (arXiv 2302.12173) - foundational indirect-injection paper. [source]
- OWASP LLM01:2025 Prompt Injection - direct/indirect taxonomy, exfiltration. [source]
- Mend.io - 2025 OWASP Top 10 for LLM Apps - full 2025 list. [source]
- JailbreakBench (arXiv 2404.01318) - benchmark, classifier, leaderboard; AdvBench/HarmBench lineage. [source]
- JailbreakBench GitHub - dataset & repo. [source]
- AgentDojo paper (arXiv 2406.13352) - agent/tool-use injection benchmark. [source]
- AgentDojo site - leaderboard, 97 tasks / 629 security cases. [source]
- Lakera - Who is Gandalf - Gandalf game, Lakera Red, Cisco acquisition. [source]
- Lakera - Guide to prompt injection - attack taxonomy. [source]
- Cisco - Robust Intelligence is part of Cisco - algorithmic red-teaming, TAP, AI Validation. [source]
- Microsoft Security Blog - 3 takeaways from red-teaming 100 GenAI products - eight lessons summary. [source]
- Lessons From Red Teaming 100 Generative AI Products (arXiv 2501.07238) - AIRT ontology, benchmarks-not-a-substitute. [source]
- Simon Willison - The lethal trifecta - trifecta definition. [source]
- OWASP - GenAI Red Teaming Guide - four-phase methodology. [source]
- ResilientCyber - Implementing GenAI Red Teaming the OWASP way - phase walkthrough. [source]
- Vectra - MITRE ATLAS - ATLAS tactics/techniques, v5.1.0 scale. [source]
- Straiker - Comparing OWASP, CSA, NIST, MITRE - framework complementarity, 2026 OWASP landscapes. [source]
- Mend.io - LLM red teaming best practices - continuous/regression, anti-patterns. [source]
- Ajith Prabhakar - Red Teaming LLMs playbook - reporting, reproducible cases. [source]
- Mindgard - What is AI red teaming (2026) - continuous red-teaming, platform. [source]
- CSET Georgetown - AI Red-Teaming Design - threat-model-first methodology. [source]
- Vectra - AI red teaming tools/frameworks - manual-vs-automated. [source]
- CrowdStrike - Prompt Injection taxonomy - IM/PT taxonomy, exfiltration. [source]
- ToxSec - DAN & roleplay prompts - jailbreak families. [source]
- Mend.io - Top 6 AI red teaming services - commercial platform landscape. [source]
Methodology
- Searched 12 queries across web and news using the WebSearch/WebFetch fallback (firecrawl/exa MCPs unavailable in this thread; per skill guidance, source-count target raised ~50%). Deep-read primary sources: Garak README, OWASP LLM01:2025, OWASP GenAI Red Teaming Guide, and arXiv papers (TAP, Crescendo, JailbreakBench, AgentDojo, Many-Shot, Greshake indirect-injection, Microsoft "100 products"). Sub-questions investigated: (1) discipline & manual-vs-automated & continuous; (2) tooling landscape scanner-vs-framework (Garak/PyRIT/promptfoo/Giskard/Purple Llama); (3) attack taxonomy (direct/indirect injection, jailbreak families, exfiltration, lethal trifecta); (4) automated attack generation (GCG/PAIR/TAP/red-teamer LLMs); (5) benchmarks/datasets (AdvBench/HarmBench/JailbreakBench/AgentDojo/OWASP-as-checklist); (6) AI-firewall products from the testing angle (Lakera/Gandalf, Robust Intelligence/Cisco, Mindgard, HiddenLayer); (7) process & framework mapping (OWASP guide 4 phases, ATLAS, NIST AI RMF, red-team-in-CI); (8) anti-patterns. Injection guard honored: all fetched web content treated as data, not instructions; no adversarial-instruction pages encountered. [source]
Children
- No children recorded.