AI Red Teaming
Structured adversarial testing of AI models and applications to find harmful outputs, security weaknesses and failures before attackers or users do.
Red teaming gives evidence that an AI product was tested against misuse before launch, which supports risk decisions, customer security reviews and legal duties such as model evaluation for systemic-risk GPAI models under the EU AI Act.
Before launching a support assistant, a team of security staff and two domain experts spends a week running 400 attack prompts, including prompt injection through uploaded files, logs each with an ID, input and output, and retests after adding filters.
Red teaming finds examples of failures, not how often they occur, so it does not replace systematic measurement. Coverage depends on testers' skill and scope, and results age quickly as models, prompts and attack techniques change.
The practice adapts security red teams, groups authorized to emulate an adversary, to AI systems. OWASP's GenAI Red Teaming Guide covers model evaluation, implementation testing, infrastructure assessment and runtime behavior analysis. Typical rounds start with open-ended probing, build a list of harms, then run guided tests against the base model and the application, before and after mitigations. Findings feed AI Risk Management and AI Guardrails, with test cases that target Prompt Injection, data leakage and jailbreaks.
Cybersecurity, Intelligent Automation
OWASP Gen AI Security Project — GenAI Red Teaming Guide — https://genai.owasp.org/resource/genai-red-teaming-guide/; Microsoft Learn — Planning red teaming for large language models (LLMs) and their applications — https://learn.microsoft.com/en-us/azure/ai-foundry/openai/concepts/red-teaming