2025年生成式AI红队百次测试经验白皮书-微软（英）_21页_1mb

Summary of Report: Lessons from Red Teaming 100 Generative AI Products

Microsoft’s AI Red Team has conducted red teaming operations on over 100 generative AI (GenAI) products, developing a threat model ontology to probe safety and security risks. This report outlines key insights and eight main lessons derived from these experiences, including practical recommendations and case studies.

Key findings include the necessity of understanding system capabilities and deployment contexts to identify real-world risks, the effectiveness of simple attack strategies over complex gradient-based methods, and the distinction between red teaming and safety benchmarking. Automation, such as Microsoft's open-source framework PyRIT, helps scale operations, but human judgment remains essential for creative and contextual assessments. Responsible AI harms, like generating hazardous content, are pervasive and challenging to measure, while large language models (LLMs) amplify existing security risks, introducing new attack vectors. Securing AI systems is an ongoing process due to evolving threats.

Case studies illustrate the application of these lessons, such as jailbreaking vision models to generate illegal content and assessing LLM vulnerabilities for scams. Open questions highlight the need for evolving practices to address novel capabilities, multilingual contexts, and standardized methodologies. Overall, red teaming should focus on end-to-end system risks and integrate with business and regulatory considerations to mitigate GenAI harms effectively.