2026当AI构建AI_AI研发自动化研讨会成果报告_32页_1mb

报告摘要

Workshop Report Summary: When AI Builds AI

Core Content

This report presents findings from a workshop hosted by the Center for Security and Emerging Technology (CSET) in July 2025, focusing on the automation of AI research and development (R&D). The goal was to explore how AI is currently being used to accelerate AI R&D, how this might evolve, and what the implications could be for society and policy.

Main Findings

1. AI R&D Automation as a Source of Strategic Surprise

Key Point: Increasing automation of AI R&D could lead to significant strategic surprises, with potential for extreme risks.
Expert Disagreement: While experts disagree on the likelihood of such scenarios, they agree that these possibilities warrant preparatory action.
Risk Factors: Reduced human oversight and faster AI progress could result in unforeseen consequences, such as loss of control over AI systems or rapid, unmonitored advancements.

2. Current Use of AI in AI R&D

Practical Application: Frontier AI companies are already using AI to accelerate their R&D processes, especially in engineering tasks like coding and data processing.
Examples: AI tools like Claude Code are used to speed up diagnosis, debugging, and data management. These tools are often used internally before being released publicly.
Early Adoption: Researchers frequently begin using new AI models before they are publicly available, indicating a trend of increasing internal reliance on AI.

3. Divergent Expert Views on AI R&D Automation

Different Trajectories: Experts have varying expectations about the future of AI R&D automation, ranging from rapid acceleration to plateauing progress.
Underlying Assumptions: These views are based on different mental models about how automation will proceed and what bottlenecks might exist.
Challenges in Forecasting: New data may not be sufficient to resolve conflicting views, making it hard to detect or rule out extreme scenarios like an intelligence explosion.

4. Importance of Indicators for AI R&D Automation

Need for Data: Existing empirical data is insufficient to measure, understand, or forecast the trajectory of AI R&D automation.
Proposed Indicators:
- Metrics for Broad AI Capabilities
- Benchmarks for AI R&D-Specific Capabilities
- Signs of Automation Progress within Companies
Transparency: Current transparency efforts are limited and patchy, with most information coming from voluntary company releases.

5. Policy Implications and Options

Transparency Mandates: Policymakers could consider transparency measures to better monitor AI R&D automation, even though existing mandates do not focus on this area.
Preparatory Actions: Given the potential for rapid and uncontrolled AI progress, there is a need for policy interventions to increase visibility and oversight.
Options: Policies could include improved data collection, monitoring of automation progress, and greater transparency in AI R&D processes.

Key Questions and Considerations

How much will AI R&D progress? Will it accelerate due to compounding improvements or plateau due to diminishing returns?
What is the ceiling of AI capabilities? Can AI surpass human performance in all key areas, or will there be inherent limitations?
What are the bottlenecks? Potential limitations include:
- Hard-to-automate tasks: Some tasks may not be easily handled by AI, such as those requiring creativity or complex decision-making.
- Last-mile data: Access to real-world data may limit AI's ability to impact non-AI domains.
- Computational power: The availability of compute resources could constrain the rate of AI R&D automation.

Conclusion

The automation of AI R&D is already occurring and is likely to increase.
The trajectory of this automation is uncertain, with significant implications for both the development of AI and its societal impact.
Better data collection and transparency mechanisms are needed to understand and manage the risks associated with AI R&D automation.
Policymakers should consider a range of options to increase visibility and oversight, especially as the potential for rapid, uncontrolled progress grows.

Takeaways

Strategic Surprise: AI R&D automation could lead to unexpected and extreme outcomes.
Current Use: AI is already used to accelerate AI R&D, particularly in engineering tasks.
Expert Disagreement: Views on the future of AI R&D automation are divided, making forecasting difficult.
Need for Indicators: Systematic collection of indicators is essential for tracking progress and understanding risks.
Policy Actions: Transparency and data access are key areas for policy intervention to manage potential risks.

展开完整摘要