2024-10-13-OpenAI-_OpenAI+o1大模型_英文技术报告_43页_1mb

OpenAI o1 Model Summary Report

Introduction

OpenAI's o1 model series incorporates large-scale reinforcement learning and chain-of-thought (CoT) reasoning to enhance both capabilities and safety. These models, including o1-preview and o1-mini, demonstrate superior performance in generating safe responses by adhering to safety policies, such as refusing harmful content and resisting jailbreaks. However, this increased intelligence also introduces risks, necessitating robust alignment methods, external red teaming, and ongoing safety protocols.

Model Training and Capabilities

• Trained using reinforcement learning, o1 models generate CoT before responding, improving reasoning in tasks like coding, math, and safety benchmarks.
• Data sources include publicly available datasets, proprietary partnerships, and custom datasets, with rigorous filtering to mitigate risks like harmful content.
• o1-preview excels as a general model, while o1-mini is optimized for speed and coding tasks.

Safety Enhancements and Evaluations

• Disallowed Content and Jailbreak Resistance: o1 models show near-perfect refusal on harmful prompts and significantly improved performance in resisting known jailbreaks compared to GPT-4o.
• Hallucination and Bias: Reduction in hallucinations on certain benchmarks, with better performance than some peers; slight medium risk in persuasion and biological threat creation under the Preparedness Framework.
• Chain-of-Thought Safety: CoT summaries are monitored for deceptive behavior, but risk of faithful monitoring remains an issue. Hallucinations increased in complexity, potentially misleading users.

Risk Assessment and Mitigations

• Overall risk rated as medium in key areas like persuasion and biological threat creation, with low risk in model autonomy.
• External evaluations and red teaming confirmed strengths in reasoning but highlighted risks in offensive cybersecurity, chemistry, and biosafety tasks.
• Multilingual performance is improved, with o1-mini outperforming GPT-4o in various languages.

Conclusion

The o1 models bring advanced reasoning capabilities that enhance safety and performance, but also escalate certain risks. OpenAI proceeds with caution, incorporating safety mitigations and committing to iterative deployment to improve model safety. Deployment balances benefits and risks, with ongoing research into CoT monitoring and risk management.