Research brief / Adversarial AI
Test the task, not just the refusal
Evaluate legitimate completion, policy violations, and scorer disagreement separately when testing an AI application.
The question
How do you distinguish a useful mitigation from a model that simply stops doing its job?
Source reports
Research benchmark and environment for evaluating prompt-injection attacks and defenses against LLM agents, with configurable task suites, attacks, and defenses.
Open-source framework for generative-AI risk assessment; its documentation describes single- and multi-turn strategies, scenario composition, memory, and configurable scoring.
Scanner that combines static, dynamic, and adaptive probes for failures including data leakage, hallucination, prompt injection, misinformation, toxicity, and jailbreaks.
CLI and library for evaluating and red-teaming LLM applications, comparing models, and automating checks in CI/CD.
Living knowledge base of adversary tactics and techniques involving AI, grounded in real-world observations and realistic demonstrations, including attacks on AI-enabled systems and abuse of AI capabilities.
RedLatent interpretation
A defense that blocks every request can look safe under a failure-only metric. Keep a clean-task control and an independent adjudication path. A detector hit is a triage signal; disagreement with a human reviewer is information about the measurement, not something to discard.
Practical relevance
Map the application’s assets and trust boundaries to relevant ATLAS techniques. Use synthetic documents and canaries, with clean, adversarial, and defended task conditions. Have reviewers judge a sample without seeing the automated score first.
Limitations
- Simulated tasks do not prove production security; no scanner establishes safety through an absence of hits.
- Provider calls, telemetry, and transcripts may leave the local machine even if orchestration runs locally. No attack or mitigation was executed here.
Proposed validation
Does the mitigation reduce independently confirmed violations while retaining legitimate task completion on the same corpus?
A research question, not an executed experiment.
Source bibliography
- Python Risk Identification Tool for generative AI (PyRIT) ↗
Microsoft
Publication date: Date not supplied · Editorial check 2026-10-26 UTC
Project README / official documentation; source-led editorial check, no execution - garak, LLM vulnerability scanner ↗
NVIDIA / garak maintainers
Publication date: Date not supplied · Editorial check 2026-10-26 UTC
Project README / official documentation; source-led editorial check, no execution - Promptfoo: LLM evals & red teaming ↗
Promptfoo maintainers; repository states Promptfoo is part of OpenAI
Publication date: Date not supplied · Editorial check 2026-10-26 UTC
Project README / official documentation; source-led editorial check, no execution - MITRE ATLAS™ ↗
MITRE
Publication date: Date not supplied · Editorial check 2026-10-26 UTC
Project README / official documentation; source-led editorial check, no execution - AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agents ↗
ETH Zurich and Invariant Labs research authors
Publication date: Date not supplied · Editorial check 2026-10-26 UTC
Project README / official documentation; source-led editorial check, no execution