AI Red-Teaming and Security-Testing Tooling
Parent: AI Agent Ecosystems · Published reference · snapshot 2026-09-08 · skill ai-agent-engineering
Also known as: AI Red-Teaming & Security-Testing Tooling for LLM Apps (2024-2026)
↓ Facts as markdownall context files
AI red-teaming is the offensive testing discipline for LLM and generative-AI applications: systematically generating adversarial inputs to find where a model or app fails in ways we don't want (jailbr
These notes link each claim to its source. Some citations identify archived research reports that are no longer hosted on this site. A published reference means the content is available; it does not certify independent review or accuracy.Read the editorial policy and follow the sources before relying on a claim.
Overview
- AI red-teaming is the offensive testing discipline for LLM and generative-AI applications: systematically generating adversarial inputs to find where a model or app fails in ways we don't want (jailbreaks, prompt injection, data/system-prompt exfiltration, harmful-content generation, insecure tool use), then feeding those findings back into fixes and regression tests. It is distinct from runtime guardrails/defense (input/output filters, dual-LLM/CaMeL), which are the defensive counterpart and are covered by the existing agent-reliability-guardrails reference - this report focuses on the testing tooling and methodology and cross-references the defensive layer where relevant. [archived source: https://llms-explorer.com/sources/global-ai-hub/ai-redteaming-tooling/#overview]
- By 2025-2026 the field has matured from ad-hoc manual probing into a tooled discipline with: (1) open-source scanners and frameworks (Garak, PyRIT, promptfoo, Giskard, Purple Llama); (2) standardized benchmarks (HarmBench, JailbreakBench, AdvBench, AgentDojo); (3) automated attack-generation algorithms (GCG, PAIR, TAP, Crescendo, many-shot); (4) commercial continuous red-teaming platforms (Lakera/Cisco, Mindgard, HiddenLayer, Robust Intelligence/Cisco AI Defense); and (5) governance/process frameworks (OWASP LLM Top 10 2025, OWASP GenAI Red Teaming Guide, MITRE ATLAS, NIST AI RMF). The dominant consensus lesson is that automation augments but does not replace human red-teamers, and that red-teaming must be continuous, not one-off (Microsoft Security Blog; Mend.io). [archived source: https://llms-explorer.com/sources/global-ai-hub/ai-redteaming-tooling/#overview]
1. AI Red-Teaming as a Discipline (offensive testing) — *Confidence: High*
- Definition & posture. Red-teaming starts by defining a threat model: a description of the AI system, the relevant vulnerabilities, and the contexts in which they arise, including human interactions (CSET Georgetown). It is offensive testing, distinct from blue-team runtime defenses. [archived source: https://llms-explorer.com/sources/global-ai-hub/ai-redteaming-tooling/#1-ai-red-teaming-as-a-discipline-offensive-testing-confidence-high]
- Manual vs automated. Automated tools (PyRIT, Garak, commercial platforms) excel at systematic coverage and regression testing but cannot match human creativity in discovering novel attacks; manual expert testing remains essential (Mend.io; Vectra). Microsoft's own framing: PyRIT "is not a replacement for manual red teaming... it augments an AI red teamer's existing domain expertise and automates the tedious tasks" (Microsoft Security Blog, 2024). [archived source: https://llms-explorer.com/sources/global-ai-hub/ai-redteaming-tooling/#1-ai-red-teaming-as-a-discipline-offensive-testing-confidence-high]
- Continuous red-teaming. The 2025-2026 shift is toward continuous adversarial testing run regularly in staging or production monitoring, rather than periodic one-shot engagements (Ajith Prabhakar playbook; Mindgard). Regression testing after fixes is treated as mandatory, not optional (Mend.io). [archived source: https://llms-explorer.com/sources/global-ai-hub/ai-redteaming-tooling/#1-ai-red-teaming-as-a-discipline-offensive-testing-confidence-high]
- Microsoft's lessons from 100 products. Eight key lessons include: start from downstream impact rather than attack mechanics; prompt engineering often outperforms gradient-based attacks in practice; and red-teaming cannot be replaced by safety benchmarks. Their AIRT ontology models a finding as: an Actor conducts an Attack leveraging TTPs to exploit a Weakness in a System, creating an Impact (Lessons From Red Teaming 100 Generative AI Products, arXiv 2501.07238; Microsoft Security Blog). [archived source: https://llms-explorer.com/sources/global-ai-hub/ai-redteaming-tooling/#1-ai-red-teaming-as-a-discipline-offensive-testing-confidence-high]
2. Tooling Landscape: Scanners vs Frameworks — *Confidence: High*
- The landscape splits into scanners (run a fixed/curated battery of probes, give a vulnerability report) and frameworks (composable building blocks you script your own attacks with). The major open-source tools: [archived source: https://llms-explorer.com/sources/global-ai-hub/ai-redteaming-tooling/#2-tooling-landscape-scanners-vs-frameworks-confidence-high]
Garak (NVIDIA) — *the leading open-source LLM vulnerability scanner*
- Name = "Generative AI Red-teaming & Assessment Kit." Developed Spring 2023 by Prof. Leon Derczynski; homed at NVIDIA as an open-source project with long-term support since Nov 2024 (Wikipedia); GitHub). [archived source: https://llms-explorer.com/sources/global-ai-hub/ai-redteaming-tooling/#garak-nvidia-the-leading-open-source-llm-vulnerability-scanner]
- Five-component plugin architecture: Probes (generate adversarial interactions), Detectors (identify failure modes in responses), Generators (interface to model platforms - OpenAI, HuggingFace, Bedrock, etc.), Harnesses (orchestrate the workflow; default probewise), and Evaluators (produce findings reports) (Garak README). [archived source: https://llms-explorer.com/sources/global-ai-hub/ai-redteaming-tooling/#garak-nvidia-the-leading-open-source-llm-vulnerability-scanner]
- 20+ probe modules covering prompt injection (encoding-based, structural), DAN-family jailbreaks, encoding/Base64 guardrail bypass, data leakage, package/dependency hallucination, toxicity, malware generation, XSS via LLM output, and glitch tokens. Notable modules: encoding, dan, promptinject, malwaregen, realtoxicityprompts (Garak README; Databricks blog). [archived source: https://llms-explorer.com/sources/global-ai-hub/ai-redteaming-tooling/#garak-nvidia-the-leading-open-source-llm-vulnerability-scanner]
- CLI-driven: garak --target_type openai --target_name <model> --probes encoding. License Apache 2.0 (Garak README). [archived source: https://llms-explorer.com/sources/global-ai-hub/ai-redteaming-tooling/#garak-nvidia-the-leading-open-source-llm-vulnerability-scanner]
Microsoft PyRIT — *automation framework, "Metasploit for LLMs"*
- PyRIT = Python Risk Identification Toolkit. Released Feb 2024 (Microsoft Security Blog; The Hacker News). [archived source: https://llms-explorer.com/sources/global-ai-hub/ai-redteaming-tooling/#microsoft-pyrit-automation-framework-metasploit-for-llms]
- Four core components: Targets (the AI under test), Converters (transform/obfuscate prompts before sending), Scorers (evaluate whether responses are harmful), Orchestrators (manage multi-turn conversation flow). It is adaptive - it changes tactics based on the system's response and generates the next input until the goal is reached (Medium deep dive; ToxSec). [archived source: https://llms-explorer.com/sources/global-ai-hub/ai-redteaming-tooling/#microsoft-pyrit-automation-framework-metasploit-for-llms]
- In one Copilot exercise, Microsoft generated several thousand malicious prompts and scored outputs "in hours instead of weeks." Now integrated into Azure AI Foundry's AI Red Teaming Agent (Microsoft Security Blog; Microsoft Learn). [archived source: https://llms-explorer.com/sources/global-ai-hub/ai-redteaming-tooling/#microsoft-pyrit-automation-framework-metasploit-for-llms]
promptfoo — *eval + red-team, CI-native*
- CLI/library for evaluating and red-teaming LLM apps, runs 100% locally; used by OpenAI and Anthropic and 156 of the Fortune 500 (GitHub; Promptfoo docs). [archived source: https://llms-explorer.com/sources/global-ai-hub/ai-redteaming-tooling/#promptfoo-eval-red-team-ci-native]
- Built-in vulnerability scanner covering 50+ vulnerability types and OWASP LLM Top 10, with presets aligned to NIST AI RMF, OWASP LLM Top 10, and MITRE ATLAS (Promptfoo MITRE ATLAS; Promptfoo config). [archived source: https://llms-explorer.com/sources/global-ai-hub/ai-redteaming-tooling/#promptfoo-eval-red-team-ci-native]
- CI/CD integration: can automatically fail builds if AI components regress or become vulnerable - the canonical "red-team-in-CI" pattern (Promptfoo CI/CD). [archived source: https://llms-explorer.com/sources/global-ai-hub/ai-redteaming-tooling/#promptfoo-eval-red-team-ci-native]
Giskard — *open-source scan for security + quality*
- Apache-2.0 Python library that auto-detects security vulns (prompt injection, data leakage) and quality issues (hallucination, bias, sycophancy) in LLMs, RAG systems, and classic ML (Giskard docs; Giskard site). [archived source: https://llms-explorer.com/sources/global-ai-hub/ai-redteaming-tooling/#giskard-open-source-scan-for-security-quality]
- Uses a mixture of predefined examples, heuristics, and LLM-assisted detectors (GPT-4-based); notable for sycophancy testing to surface hallucination without ground truth (Giskard docs; HuggingFace blog). [archived source: https://llms-explorer.com/sources/global-ai-hub/ai-redteaming-tooling/#giskard-open-source-scan-for-security-quality]
Meta Purple Llama — *guardrail models + the offensive benchmark*
- A toolset spanning Llama Guard (input/output content moderation classifier), Prompt Guard (mDeBERTa-based classifier for jailbreaks + indirect injection), and CyberSecEval (GitHub; Meta Prompt Guard overview). [archived source: https://llms-explorer.com/sources/global-ai-hub/ai-redteaming-tooling/#meta-purple-llama-guardrail-models-the-offensive-benchmark]
- For the testing angle, CyberSecEval is the key piece: a benchmark of an LLM's propensity to (a) generate insecure code and (b) comply with cyberattack assistance. CyberSecEval 3 added visual prompt-injection tests, spear-phishing capability tests, and autonomous offensive cyber-operations tests (CyberSecEval paper, arXiv 2312.04724; Meta AI research). (Llama Guard / Prompt Guard themselves are defensive - cross-reference agent-reliability-guardrails.) [archived source: https://llms-explorer.com/sources/global-ai-hub/ai-redteaming-tooling/#meta-purple-llama-guardrail-models-the-offensive-benchmark]
3. Attack Taxonomy — *Confidence: High*
- Direct vs indirect prompt injection. Direct injection (often conflated with jailbreaking) is when a user's input overrides/reveals the system prompt. Indirect injection smuggles malicious instructions via untrusted content the model ingests (RAG docs, web pages, files, tool outputs) - here the attacker is not the user but a third party poisoning the data channel (OWASP LLM01:2025; Greshake et al., arXiv 2302.12173 - the foundational indirect-injection paper; CrowdStrike). [archived source: https://llms-explorer.com/sources/global-ai-hub/ai-redteaming-tooling/#3-attack-taxonomy-confidence-high]
- Jailbreak families. Documented families include DAN ("Do Anything Now") roleplay/dual-persona, generic roleplay/persona-escalation and "admin" status, payload splitting, obfuscation/encoding (Base64, multilingual injection, hiding in code comments), and multi-turn gradual escalation (ToxSec DAN; OnSecurity; Astra guide). [archived source: https://llms-explorer.com/sources/global-ai-hub/ai-redteaming-tooling/#3-attack-taxonomy-confidence-high]
- Many-shot jailbreaking (Anthropic, 2024). Prompt the model with hundreds-to-thousands of in-context examples of undesirable behavior, then ask the harmful question; effectiveness follows a power law with shot count and is newly feasible due to long context windows (Anthropic Many-Shot paper PDF; Prompt Security explainer). [archived source: https://llms-explorer.com/sources/global-ai-hub/ai-redteaming-tooling/#3-attack-taxonomy-confidence-high]
- Crescendo (multi-turn). A seemingly benign multi-turn attack that exploits the model's tendency to follow patterns and focus on its own recent output; reaches up to 100% ASR on many tasks and beats prior SOTA by 29-71% on AdvBench subsets (Crescendo paper, arXiv 2404.01833; Crescendo site). [archived source: https://llms-explorer.com/sources/global-ai-hub/ai-redteaming-tooling/#3-attack-taxonomy-confidence-high]
- Data / system-prompt exfiltration. A successful injection in a tool-enabled app can exfiltrate data directly via tools (e.g., writing to a public GitHub repo, DNS exfil, markdown-image rendering) - this is LLM02 Sensitive Information Disclosure and LLM07 System Prompt Leakage in OWASP 2025 (CrowdStrike indirect-injection blog; OWASP LLM01:2025). [archived source: https://llms-explorer.com/sources/global-ai-hub/ai-redteaming-tooling/#3-attack-taxonomy-confidence-high]
- The lethal trifecta (cross-ref to guardrails). Simon Willison's June 2025 framing: an agent is structurally insecure when it combines (1) access to private data, (2) exposure to untrusted content, and (3) the ability to communicate externally - a single poisoned input can then exfiltrate data with no traditional code vuln. As a testing target this maps directly to indirect-injection + tool-poisoning test cases; the defensive mitigation is covered by agent-reliability-guardrails (Simon Willison; Promptfoo lethal-trifecta testing; HiddenLayer). [archived source: https://llms-explorer.com/sources/global-ai-hub/ai-redteaming-tooling/#3-attack-taxonomy-confidence-high]
4. Automated Attack Generation — *Confidence: High*
- GCG (Greedy Coordinate Gradient). Token-level, white-box - optimizes an adversarial suffix via greedy coordinate gradient descent to force affirmative ("Sure, here is…") outputs. High success but orders of magnitude more queries than search-based methods; suffixes can transfer across models (Tree of Attacks paper comparison, arXiv 2312.02119; EmergentMind GCG+PAIR). [archived source: https://llms-explorer.com/sources/global-ai-hub/ai-redteaming-tooling/#4-automated-attack-generation-confidence-high]
- PAIR (Prompt Automatic Iterative Refinement). Uses an attacker LLM steered by meta-prompts to iteratively refine semantically coherent adversarial prompts - black-box, query-efficient (TAP paper). [archived source: https://llms-explorer.com/sources/global-ai-hub/ai-redteaming-tooling/#4-automated-attack-generation-confidence-high]
- TAP (Tree of Attacks with Pruning). Black-box; an attacker LLM iteratively refines a tree of candidate prompts with pruning. Against GPT-4o it finds jailbreaks for 16% more prompts than PAIR with 60% fewer queries. This is the algorithm Robust Intelligence/Cisco productized for algorithmic red-teaming (TAP paper, arXiv 2312.02119; Cisco/Robust Intelligence). [archived source: https://llms-explorer.com/sources/global-ai-hub/ai-redteaming-tooling/#4-automated-attack-generation-confidence-high]
- Red-teamer LLMs & fuzzing. The common modern pattern is an attacker/red-teamer LLM that generates, mutates, and scores attacks in a loop - the core of PyRIT orchestrators, TAP, PAIR, and Crescendo automation ("Crescendomation"). Hybrid GCG+PAIR combines gradient suffixes with semantic refinement for up to +33pp ASR over single-strategy (EmergentMind; Crescendo paper). [archived source: https://llms-explorer.com/sources/global-ai-hub/ai-redteaming-tooling/#4-automated-attack-generation-confidence-high]
5. Benchmarks & Datasets — *Confidence: High*
- AdvBench (Zou et al., 2023) - the original harmful-behaviors/harmful-strings dataset introduced with GCG; now a component of later benchmarks (JailbreakBench paper, arXiv 2404.01318). [archived source: https://llms-explorer.com/sources/global-ai-hub/ai-redteaming-tooling/#5-benchmarks-datasets-confidence-high]
- HarmBench (Center for AI Safety, 2024) - standardized evaluation framework for automated red-teaming and robust refusal, broader topic coverage including copyright and multimodal (JailbreakBench paper). [archived source: https://llms-explorer.com/sources/global-ai-hub/ai-redteaming-tooling/#5-benchmarks-datasets-confidence-high]
- JailbreakBench (NeurIPS 2024) - open robustness benchmark: 100 harmful + 100 benign behaviors across 10 categories aligned to OpenAI usage policies, a validated jailbreak classifier, an artifact repository, and a public leaderboard for attacks and defenses (JailbreakBench GitHub; paper, arXiv 2404.01318). [archived source: https://llms-explorer.com/sources/global-ai-hub/ai-redteaming-tooling/#5-benchmarks-datasets-confidence-high]
- AgentDojo (2024) - the key benchmark for tool-use / agent injection: a dynamic environment with 97 realistic tasks and 629 security test cases across banking, Slack, travel, and workspace domains; metrics are benign utility, utility-under-attack, and attack success rate. Used by US/UK AISI to demonstrate Claude 3.5 Sonnet's injection vulnerability (AgentDojo paper, arXiv 2406.13352; AgentDojo site). [archived source: https://llms-explorer.com/sources/global-ai-hub/ai-redteaming-tooling/#5-benchmarks-datasets-confidence-high]
- OWASP LLM Top 10 (2025) as a testing checklist. Functions as a structured 10-category test plan for any production LLM deployment (Mend.io 2025 guide): LLM01 Prompt Injection, LLM02 Sensitive Information Disclosure, LLM03 Supply Chain, LLM04 Data & Model Poisoning, LLM05 Improper Output Handling, LLM06 Excessive Agency, LLM07 System Prompt Leakage, LLM08 Vector & Embedding Weaknesses, LLM09 Misinformation, LLM10 Unbounded Consumption (OWASP LLM01:2025; Oligo). [archived source: https://llms-explorer.com/sources/global-ai-hub/ai-redteaming-tooling/#5-benchmarks-datasets-confidence-high]
6. AI Firewalls / Runtime-Defense Products from the Testing Angle — *Confidence: High*
- These are defensive products, but each ships a red-team/testing capability or dataset relevant to offensive testing: [archived source: https://llms-explorer.com/sources/global-ai-hub/ai-redteaming-tooling/#6-ai-firewalls-runtime-defense-products-from-the-testing-angle-confidence-high]
- Lakera - Gandalf & Lakera Red. Gandalf is a gamified prompt-injection challenge (extract a secret password through 7-8 escalating defense levels); it has drawn 1M+ players and 80M+ adversarial prompts that feed Lakera's threat intel. Lakera Red runs automated attack simulations against LLM apps pre-production. Lakera (incl. Guard, Red, Gandalf dataset) was acquired by Cisco in 2025 (Lakera "Who is Gandalf"; Lakera prompt-injection guide). [archived source: https://llms-explorer.com/sources/global-ai-hub/ai-redteaming-tooling/#6-ai-firewalls-runtime-defense-products-from-the-testing-angle-confidence-high]
- Robust Intelligence → Cisco AI Defense. Robust Intelligence (Harvard spinout, founded 2019) built what it calls the first algorithmic red-teaming solution, using TAP to jailbreak LLMs in seconds rather than weeks of human effort. Acquired by Cisco Oct 2024; now powers Cisco AI Defense "AI Validation" (Cisco; Cisco blog). [archived source: https://llms-explorer.com/sources/global-ai-hub/ai-redteaming-tooling/#6-ai-firewalls-runtime-defense-products-from-the-testing-angle-confidence-high]
- Mindgard. Automated AI red-teaming / DAST-AI platform for LLMs, agents, multimodal; Lancaster University spinout (2022), recognized in the OWASP GenAI Security Solutions Landscape; emphasizes continuous red-teaming (Mindgard; Mindgard/S&P Global). [archived source: https://llms-explorer.com/sources/global-ai-hub/ai-redteaming-tooling/#6-ai-firewalls-runtime-defense-products-from-the-testing-angle-confidence-high]
- HiddenLayer & Prompt Guard. HiddenLayer's AISec platform focuses on ML model protection and threat intel; Meta's Prompt Guard classifier is the open-source guardrail you test against (Mend.io top-6 services; Meta Prompt Guard). [archived source: https://llms-explorer.com/sources/global-ai-hub/ai-redteaming-tooling/#6-ai-firewalls-runtime-defense-products-from-the-testing-angle-confidence-high]
7. Process: Scoping, CI, Reporting, and Framework Mapping — *Confidence: High*
- OWASP GenAI Red Teaming Guide (Jan 2025) structures red-teaming into four phases: (1) model evaluation (provenance, data pipelines), (2) implementation testing (guardrails in place), (3) infrastructure/system assessment (deployed exploitable components), and (4) runtime behavior analysis (business processes and multi-component interactions in production) (OWASP GenAI Red Teaming Guide; ResilientCyber walkthrough; CSO Online). [archived source: https://llms-explorer.com/sources/global-ai-hub/ai-redteaming-tooling/#7-process-scoping-ci-reporting-and-framework-mapping-confidence-high]
- Three governance frameworks, three lifecycle phases. The standard mapping: OWASP LLM Top 10 for the development/secure-coding phase, MITRE ATLAS for operations/threat-modeling/detection, NIST AI RMF (Govern/Map/Measure/Manage) for governance/compliance - they complement, not compete (Straiker; Medium - Elias Silva). [archived source: https://llms-explorer.com/sources/global-ai-hub/ai-redteaming-tooling/#7-process-scoping-ci-reporting-and-framework-mapping-confidence-high]
- MITRE ATLAS scale. As of the Nov 2025 v5.1.0 update: 16 tactics, 84 techniques, 32 mitigations, 42 case studies, with continued additions of agentic-AI techniques into 2026 (Vectra; Repello). OWASP published a first-ever Red Teaming Landscape in 2026 alongside separate LLM/GenAI and agentic-AI landscapes (Straiker). [archived source: https://llms-explorer.com/sources/global-ai-hub/ai-redteaming-tooling/#7-process-scoping-ci-reporting-and-framework-mapping-confidence-high]
- Red-team in CI + reporting. promptfoo is the reference implementation: run the red-team suite in CI/CD and fail the build on regression, with reports mapped to OWASP/NIST/ATLAS (Promptfoo CI/CD). Best-practice reporting documents every scenario with goals, prompts, model versions, and observed behaviors as reproducible test cases (Ajith Prabhakar playbook). [archived source: https://llms-explorer.com/sources/global-ai-hub/ai-redteaming-tooling/#7-process-scoping-ci-reporting-and-framework-mapping-confidence-high]
8. Anti-Patterns — *Confidence: High*
- One-off red-teaming. Treating red-teaming as a single pre-launch event; models, prompts, and threat landscape drift, so testing must be continuous (Mindgard; Mend.io). [archived source: https://llms-explorer.com/sources/global-ai-hub/ai-redteaming-tooling/#8-anti-patterns-confidence-high]
- Testing only direct injection. Ignoring indirect injection / tool-poisoning (RAG docs, web content, tool outputs) - the highest-impact agentic attack surface and the core of the lethal trifecta (Greshake et al.; AgentDojo). [archived source: https://llms-explorer.com/sources/global-ai-hub/ai-redteaming-tooling/#8-anti-patterns-confidence-high]
- No regression after fixes. Failing to re-run the attack that found a bug after patching; "regression testing is not optional in AI security" (Mend.io). [archived source: https://llms-explorer.com/sources/global-ai-hub/ai-redteaming-tooling/#8-anti-patterns-confidence-high]
- Treating benchmarks as a substitute for red-teaming. Microsoft explicitly warns against the idea that safety benchmarks can replace red-teaming - benchmarks measure known behaviors; red-teaming finds novel failures (Lessons From Red Teaming 100, arXiv 2501.07238). [archived source: https://llms-explorer.com/sources/global-ai-hub/ai-redteaming-tooling/#8-anti-patterns-confidence-high]
- Over-relying on gradient attacks. In real products, prompt engineering often outperforms gradient-based attacks (GCG-style) and is far cheaper; teams that fixate on academic white-box attacks miss the easy wins (Microsoft Security Blog). [archived source: https://llms-explorer.com/sources/global-ai-hub/ai-redteaming-tooling/#8-anti-patterns-confidence-high]
- Automation-only. Assuming an automated scanner is sufficient; it cannot match human creativity for novel attacks (Vectra; Microsoft, 2024). [archived source: https://llms-explorer.com/sources/global-ai-hub/ai-redteaming-tooling/#8-anti-patterns-confidence-high]
9. Suggested Child Sub-Concepts (6-10 future concepts)
- Open-source red-team scanners - Garak, Giskard, promptfoo red-team (scanner-vs-framework, probe/detector architecture, CI integration). [archived source: https://llms-explorer.com/sources/global-ai-hub/ai-redteaming-tooling/#9-suggested-child-sub-concepts-6-10-future-concepts]
- Adversarial attack-generation algorithms - GCG, PAIR, TAP, fuzzing, red-teamer LLMs (white-box vs black-box, query efficiency, transferability). [archived source: https://llms-explorer.com/sources/global-ai-hub/ai-redteaming-tooling/#9-suggested-child-sub-concepts-6-10-future-concepts]
- Jailbreak families & prompt-injection taxonomy (offensive) - DAN/roleplay, payload splitting, encoding, many-shot, Crescendo; direct vs indirect; exfiltration patterns. [archived source: https://llms-explorer.com/sources/global-ai-hub/ai-redteaming-tooling/#9-suggested-child-sub-concepts-6-10-future-concepts]
- Agent / tool-use injection testing - AgentDojo, indirect injection, tool-poisoning, lethal-trifecta test cases (cross-ref guardrails). [archived source: https://llms-explorer.com/sources/global-ai-hub/ai-redteaming-tooling/#9-suggested-child-sub-concepts-6-10-future-concepts]
- Red-team benchmarks & datasets - AdvBench, HarmBench, JailbreakBench, CyberSecEval; leaderboards and validated classifiers. [archived source: https://llms-explorer.com/sources/global-ai-hub/ai-redteaming-tooling/#9-suggested-child-sub-concepts-6-10-future-concepts]
- PyRIT & orchestration frameworks - Targets/Converters/Scorers/Orchestrators, adaptive multi-turn automation, Azure AI Foundry Red Teaming Agent. [archived source: https://llms-explorer.com/sources/global-ai-hub/ai-redteaming-tooling/#9-suggested-child-sub-concepts-6-10-future-concepts]
- Red-team process & governance mapping - OWASP GenAI Red Teaming Guide 4 phases, OWASP LLM Top 10 as test plan, MITRE ATLAS, NIST AI RMF; red-team-in-CI and reporting. [archived source: https://llms-explorer.com/sources/global-ai-hub/ai-redteaming-tooling/#9-suggested-child-sub-concepts-6-10-future-concepts]
- Commercial continuous red-teaming platforms - Lakera/Gandalf, Cisco AI Defense (Robust Intelligence), Mindgard, HiddenLayer (DAST-AI, algorithmic red-teaming). [archived source: https://llms-explorer.com/sources/global-ai-hub/ai-redteaming-tooling/#9-suggested-child-sub-concepts-6-10-future-concepts]
Knowledge Gaps
- Quantitative tool comparisons (precision/recall of Garak vs promptfoo vs Giskard detectors on a common suite) are scarce in public sources - vendor claims dominate; treat comparative effectiveness as Low confidence. [archived source: https://llms-explorer.com/sources/global-ai-hub/ai-redteaming-tooling/#knowledge-gaps]
- Robust Intelligence post-acquisition specifics beyond the TAP-based algorithmic red-teaming and "AI Validation" branding are thin in public sources. [archived source: https://llms-explorer.com/sources/global-ai-hub/ai-redteaming-tooling/#knowledge-gaps]
- Multimodal and voice red-teaming tooling (image/audio injection) is emerging (CyberSecEval 3 visual injection; ATLAS agentic additions) but tooling maturity is unclear - flagged as a frontier, not a settled area. [archived source: https://llms-explorer.com/sources/global-ai-hub/ai-redteaming-tooling/#knowledge-gaps]
Sources
- NVIDIA/garak GitHub - the LLM vulnerability scanner repo. [archived source: https://llms-explorer.com/sources/global-ai-hub/ai-redteaming-tooling/#sources]
- Garak README (raw) - architecture, probes, CLI, Apache-2.0 license. [archived source: https://llms-explorer.com/sources/global-ai-hub/ai-redteaming-tooling/#sources]
- Garak (software) - Wikipedia) - history, ownership, recognition. [archived source: https://llms-explorer.com/sources/global-ai-hub/ai-redteaming-tooling/#sources]
- Databricks - Applying Garak - probe categories in practice. [archived source: https://llms-explorer.com/sources/global-ai-hub/ai-redteaming-tooling/#sources]
- Microsoft Security Blog - Announcing PyRIT (2024) - PyRIT launch, augments-not-replaces framing. [archived source: https://llms-explorer.com/sources/global-ai-hub/ai-redteaming-tooling/#sources]
- The Hacker News - Microsoft releases PyRIT - overview. [archived source: https://llms-explorer.com/sources/global-ai-hub/ai-redteaming-tooling/#sources]
- Medium - Automating AI Red Teaming with PyRIT - Targets/Converters/Scorers/Orchestrators. [archived source: https://llms-explorer.com/sources/global-ai-hub/ai-redteaming-tooling/#sources]
- Microsoft Learn - AI Red Teaming Agent (Foundry) - PyRIT in Azure AI Foundry. [archived source: https://llms-explorer.com/sources/global-ai-hub/ai-redteaming-tooling/#sources]
- Promptfoo - LLM red teaming docs - scanner scope, 50+ vuln types. [archived source: https://llms-explorer.com/sources/global-ai-hub/ai-redteaming-tooling/#sources]
- promptfoo/promptfoo GitHub - used by OpenAI/Anthropic, CI/CD. [archived source: https://llms-explorer.com/sources/global-ai-hub/ai-redteaming-tooling/#sources]
- Promptfoo - CI/CD integration - fail builds on regression. [archived source: https://llms-explorer.com/sources/global-ai-hub/ai-redteaming-tooling/#sources]
- Promptfoo - MITRE ATLAS preset - framework mapping. [archived source: https://llms-explorer.com/sources/global-ai-hub/ai-redteaming-tooling/#sources]
- Promptfoo - testing the lethal trifecta - trifecta as test cases. [archived source: https://llms-explorer.com/sources/global-ai-hub/ai-redteaming-tooling/#sources]
- Giskard - LLM vulnerabilities docs - scan categories. [archived source: https://llms-explorer.com/sources/global-ai-hub/ai-redteaming-tooling/#sources]
- Giskard - LLM detectors reference - sycophancy/hallucination detectors. [archived source: https://llms-explorer.com/sources/global-ai-hub/ai-redteaming-tooling/#sources]
- meta-llama/PurpleLlama GitHub - Llama Guard, Prompt Guard, CyberSecEval. [archived source: https://llms-explorer.com/sources/global-ai-hub/ai-redteaming-tooling/#sources]
- Meta - Prompt Guard overview - jailbreak + indirect-injection classifier. [archived source: https://llms-explorer.com/sources/global-ai-hub/ai-redteaming-tooling/#sources]
- CyberSecEval paper (arXiv 2312.04724) - secure-coding + cyberattack-compliance benchmark. [archived source: https://llms-explorer.com/sources/global-ai-hub/ai-redteaming-tooling/#sources]
- Tree of Attacks paper (arXiv 2312.02119) - TAP vs PAIR vs GCG comparison. [archived source: https://llms-explorer.com/sources/global-ai-hub/ai-redteaming-tooling/#sources]
- Crescendo paper (arXiv 2404.01833) - multi-turn jailbreak. [archived source: https://llms-explorer.com/sources/global-ai-hub/ai-redteaming-tooling/#sources]
- Anthropic - Many-Shot Jailbreaking (PDF) - power-law many-shot attack. [archived source: https://llms-explorer.com/sources/global-ai-hub/ai-redteaming-tooling/#sources]
- Greshake et al. - Indirect Prompt Injection (arXiv 2302.12173) - foundational indirect-injection paper. [archived source: https://llms-explorer.com/sources/global-ai-hub/ai-redteaming-tooling/#sources]
- OWASP LLM01:2025 Prompt Injection - direct/indirect taxonomy, exfiltration. [archived source: https://llms-explorer.com/sources/global-ai-hub/ai-redteaming-tooling/#sources]
- Mend.io - 2025 OWASP Top 10 for LLM Apps - full 2025 list. [archived source: https://llms-explorer.com/sources/global-ai-hub/ai-redteaming-tooling/#sources]
- JailbreakBench (arXiv 2404.01318) - benchmark, classifier, leaderboard; AdvBench/HarmBench lineage. [archived source: https://llms-explorer.com/sources/global-ai-hub/ai-redteaming-tooling/#sources]
- JailbreakBench GitHub - dataset & repo. [archived source: https://llms-explorer.com/sources/global-ai-hub/ai-redteaming-tooling/#sources]
- AgentDojo paper (arXiv 2406.13352) - agent/tool-use injection benchmark. [archived source: https://llms-explorer.com/sources/global-ai-hub/ai-redteaming-tooling/#sources]
- AgentDojo site - leaderboard, 97 tasks / 629 security cases. [archived source: https://llms-explorer.com/sources/global-ai-hub/ai-redteaming-tooling/#sources]
- Lakera - Who is Gandalf - Gandalf game, Lakera Red, Cisco acquisition. [archived source: https://llms-explorer.com/sources/global-ai-hub/ai-redteaming-tooling/#sources]
- Lakera - Guide to prompt injection - attack taxonomy. [archived source: https://llms-explorer.com/sources/global-ai-hub/ai-redteaming-tooling/#sources]
- Cisco - Robust Intelligence is part of Cisco - algorithmic red-teaming, TAP, AI Validation. [archived source: https://llms-explorer.com/sources/global-ai-hub/ai-redteaming-tooling/#sources]
- Microsoft Security Blog - 3 takeaways from red-teaming 100 GenAI products - eight lessons summary. [archived source: https://llms-explorer.com/sources/global-ai-hub/ai-redteaming-tooling/#sources]
- Lessons From Red Teaming 100 Generative AI Products (arXiv 2501.07238) - AIRT ontology, benchmarks-not-a-substitute. [archived source: https://llms-explorer.com/sources/global-ai-hub/ai-redteaming-tooling/#sources]
- Simon Willison - The lethal trifecta - trifecta definition. [archived source: https://llms-explorer.com/sources/global-ai-hub/ai-redteaming-tooling/#sources]
- OWASP - GenAI Red Teaming Guide - four-phase methodology. [archived source: https://llms-explorer.com/sources/global-ai-hub/ai-redteaming-tooling/#sources]
- ResilientCyber - Implementing GenAI Red Teaming the OWASP way - phase walkthrough. [archived source: https://llms-explorer.com/sources/global-ai-hub/ai-redteaming-tooling/#sources]
- Vectra - MITRE ATLAS - ATLAS tactics/techniques, v5.1.0 scale. [archived source: https://llms-explorer.com/sources/global-ai-hub/ai-redteaming-tooling/#sources]
- Straiker - Comparing OWASP, CSA, NIST, MITRE - framework complementarity, 2026 OWASP landscapes. [archived source: https://llms-explorer.com/sources/global-ai-hub/ai-redteaming-tooling/#sources]
- Mend.io - LLM red teaming best practices - continuous/regression, anti-patterns. [archived source: https://llms-explorer.com/sources/global-ai-hub/ai-redteaming-tooling/#sources]
- Ajith Prabhakar - Red Teaming LLMs playbook - reporting, reproducible cases. [archived source: https://llms-explorer.com/sources/global-ai-hub/ai-redteaming-tooling/#sources]
- Mindgard - What is AI red teaming (2026) - continuous red-teaming, platform. [archived source: https://llms-explorer.com/sources/global-ai-hub/ai-redteaming-tooling/#sources]
- CSET Georgetown - AI Red-Teaming Design - threat-model-first methodology. [archived source: https://llms-explorer.com/sources/global-ai-hub/ai-redteaming-tooling/#sources]
- Vectra - AI red teaming tools/frameworks - manual-vs-automated. [archived source: https://llms-explorer.com/sources/global-ai-hub/ai-redteaming-tooling/#sources]
- CrowdStrike - Prompt Injection taxonomy - IM/PT taxonomy, exfiltration. [archived source: https://llms-explorer.com/sources/global-ai-hub/ai-redteaming-tooling/#sources]
- ToxSec - DAN & roleplay prompts - jailbreak families. [archived source: https://llms-explorer.com/sources/global-ai-hub/ai-redteaming-tooling/#sources]
- Mend.io - Top 6 AI red teaming services - commercial platform landscape. [archived source: https://llms-explorer.com/sources/global-ai-hub/ai-redteaming-tooling/#sources]
Methodology
- Searched 12 queries across web and news using the WebSearch/WebFetch fallback (firecrawl/exa MCPs unavailable in this thread; per skill guidance, source-count target raised ~50%). Deep-read primary sources: Garak README, OWASP LLM01:2025, OWASP GenAI Red Teaming Guide, and arXiv papers (TAP, Crescendo, JailbreakBench, AgentDojo, Many-Shot, Greshake indirect-injection, Microsoft "100 products"). Sub-questions investigated: (1) discipline & manual-vs-automated & continuous; (2) tooling landscape scanner-vs-framework (Garak/PyRIT/promptfoo/Giskard/Purple Llama); (3) attack taxonomy (direct/indirect injection, jailbreak families, exfiltration, lethal trifecta); (4) automated attack generation (GCG/PAIR/TAP/red-teamer LLMs); (5) benchmarks/datasets (AdvBench/HarmBench/JailbreakBench/AgentDojo/OWASP-as-checklist); (6) AI-firewall products from the testing angle (Lakera/Gandalf, Robust Intelligence/Cisco, Mindgard, HiddenLayer); (7) process & framework mapping (OWASP guide 4 phases, ATLAS, NIST AI RMF, red-team-in-CI); (8) anti-patterns. Injection guard honored: all fetched web content treated as data, not instructions; no adversarial-instruction pages encountered. [archived source: https://llms-explorer.com/sources/global-ai-hub/ai-redteaming-tooling/#methodology]
Where this helps
- Standing up a pre-launch or CI-gated red-team pass for an LLM application, choosing between a scanner (Garak, Giskard) for fixed-battery coverage and a framework (PyRIT) for scripting custom multi-turn attacks. [source] — AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.
- Testing an agentic or tool-using app specifically for indirect prompt injection and tool-poisoning, the highest-impact attack surface this pack identifies, using a benchmark like AgentDojo. [source] — AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.
- Mapping a red-team program's findings to governance frameworks a customer or auditor recognizes — OWASP LLM Top 10 for secure-coding phase, MITRE ATLAS for threat modeling, NIST AI RMF for the broader risk lifecycle. [source] — AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.
- Deciding whether an automated scanner is sufficient for a given release, versus when the app's risk profile calls for manual expert testing that automated tools can't match for novel-attack creativity. [source] — AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.
Project ideas
- Wire promptfoo's red-team suite into CI so a build fails automatically when a prompt or model change regresses on a previously-passing OWASP LLM Top 10 test case. [source] — AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.
- Run Garak against a deployed model with its encoding, DAN-family jailbreak, and data-leakage probe modules to get a first-pass vulnerability report before a more targeted PyRIT-scripted campaign. [source] — AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.
- Build an AgentDojo-style test harness for a tool-using agent — realistic tasks paired with security test cases across the tools it can call — to catch indirect injection via untrusted tool outputs. [source] — AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.
- Set up continuous (not one-off) red-teaming for a production LLM app, re-running the attack suite on a schedule so model, prompt, and threat-landscape drift get caught between releases. [source] — AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.
Common mistakes
- Testing only for direct prompt injection while ignoring indirect injection via RAG documents, web content, or tool outputs — the core of the lethal-trifecta attack surface and the highest-impact class for agentic apps. [source] — AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.
- Treating red-teaming as a single pre-launch event instead of a continuous practice, even though models, prompts, and the threat landscape all drift after release. [source] — AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.
- Skipping regression testing after a fix — re-running the exact attack that found a vulnerability is the only way to confirm the patch actually closed it. [source] — AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.
- Assuming safety benchmark scores substitute for red-teaming — benchmarks measure known behaviors, while red-teaming is specifically designed to surface novel failures benchmarks don't cover. [source] — AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.
Known issues
- Quantitative comparisons between scanners (Garak vs promptfoo vs Giskard detector precision/recall on a shared suite) are scarce in public sources, so tool selection often rests on vendor claims rather than independent benchmarks. [source] — AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.
- Automated tools cannot match human creativity for discovering genuinely novel attacks — this pack's own sources are explicit that automation-only red-teaming leaves a real gap. [source] — AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.
- Multimodal and voice red-teaming tooling (image/audio injection) is still emerging and its maturity is unclear, per this pack's knowledge-gaps section — treat it as a frontier area, not a solved one. [source] — AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.
- In real products, prompt-engineering-based attacks often outperform academic white-box methods like GCG and are far cheaper to run, so teams that over-invest in gradient-attack tooling can miss the easier, higher-yield attack surface. [source] — AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.
Children
- Open-source scanners/frameworks (Garak, PyRIT, promptfoo, Giskard, CyberSecEval) (frontier)
- Attack taxonomy (direct/indirect injection, jailbreak families, many-shot, Crescendo, exfiltration) (frontier)
- Automated attack generation (GCG, PAIR, TAP, red-teamer LLMs) (frontier)
- Benchmarks (AdvBench, HarmBench, JailbreakBench, AgentDojo) (frontier)
- Commercial continuous red-teaming (Lakera/Gandalf, Cisco AI Defense, Mindgard) (frontier)
- Process & governance (OWASP GenAI Red Teaming Guide, MITRE ATLAS, NIST AI RMF, red-team-in-CI) (frontier, viability 1.63 / 5, below research gate)
Frontier under this node: Attack taxonomy (direct/indirect injection, jailbreak families, many-shot, Crescendo, exfiltration), Automated attack generation (GCG, PAIR, TAP, red-teamer LLMs), Benchmarks (AdvBench, HarmBench, JailbreakBench, AgentDojo), Commercial continuous red-teaming (Lakera/Gandalf, Cisco AI Defense, Mindgard), Open-source scanners/frameworks (Garak, PyRIT, promptfoo, Giskard, CyberSecEval), Process & governance (OWASP GenAI Red Teaming Guide, MITRE ATLAS, NIST AI RMF, red-team-in-CI) (1.63)