深度报告-2025-07-14-美国国防部-人工智能赋能系统研制试验与鉴定指南手册2025页_152页_1mb

Developmental Test and Evaluation of Artificial Intelligence-Enabled Systems Guidebook Summary

This guidebook outlines the challenges and methodologies for testing and evaluating artificial intelligence-enabled systems (AIES) within the Department of Defense (DoD). Key points include:

Executive Summary

• The DoD developed this guidebook to address the unique challenges posed by AI technologies, focusing on developmental test and evaluation (DT&E) for AI systems.
• AI changes traditional testing approaches; comprehensive testing is infeasible due to unpredictability, opacity, and high-dimensional parameter spaces.
• Emphasis is placed on early DT&E involvement, formal methods, visibility into ML models, expanded T&E interactions, and continuous monitoring post-fielding.

AI-Driven Changes in T&E Practice

• Data and Model Quality: Critical for all ML applications; includes data VV&A, model VV&A, and handling synthetic data.
• Formal Methods: Supplement empirical testing with mathematical rigor for validation.
• Modeling and Simulation: Essential for exploring scenarios not feasible in live testing.
• Visibility into ML Models: Techniques like explainable AI (XAI) to understand model behavior.
• Life Cycle Implications: From early data preparation to post-fielding drift monitoring.

Expanded Interactions for the T&E Community

• DT&E must collaborate on contracting, requirements development, CONEMP creation, and design trade-offs.
• Certification support requires addressing AI-specific risks like brittleness, adversarial vulnerabilities, and bias.
• Early involvement reduces late-stage issues by injecting mission focus into iterative development.

Key Considerations

• AIES DT&E is iterative, requiring dynamic test planning and increased use of M&S and formal methods.
• RAI mandates (Responsible AI) influence testing, emphasizing safety, governability, and traceability.
• T&E supports assurance across diverse domains like safety, cybersecurity, and interoperability.

The guidebook concludes that DT&E for AI systems must evolve to address brittleness, brittleness, and lifecycle challenges, driven by continuous interdisciplinary collaboration.