2025-03-02-国际清算银行-让人工智能代理完成一般任务（英）_19页_3mb

Summary of "Putting AI agents through their paces on general tasks"

Introduction

The paper argues that Large Language Models (LLMs) with AGI aspirations should be evaluated through comprehensive, general tasks to ensure they can handle real-world complexities and achieve reliable performance, unlike narrow, single-task evaluations.

Methodology

Authors tested current state-of-the-art LLMs (e.g., Claude CU) using popular games like Wordle, FaceQuiz, and Flashback. These tasks were chosen to assess capabilities such as self-awareness, error correction, adaptability, and integration of multiple cognitive skills, mirroring real-world challenges.

Findings

LLM agents demonstrated mixed performance:

• In Wordle, they showed resilience to errors but struggled with consistent self-correction and handling unexpected feedback, leading to premature success declarations.
• In FaceQuiz, they accurately identified individuals but missed nuanced contextual clues, affecting overall task success.
• In Flashback, they failed to adapt strategically and were inconsistent in learning from mistakes, resulting in suboptimal placements.
  Common limitations include insufficient self-assessment, inability to fully manage ancillary subtasks, and variability in error recovery, indicating gaps in current models.

Implications

Comprehensive testing is essential for identifying weaknesses before AGI deployment. AGI-aspiring LLMs should be evaluated on tasks that combine multiple cognitive abilities to ensure they can address real-world applications reliably. Initial practical use in central banks might involve copilots for tasks like data analysis, but autonomous agents could replace human roles in well-defined domains.

Conclusion

The paper advocates for moving beyond narrow task evaluations and supports the need for broad, multi-cognitive testing to advance LLMs toward AGI and superintelligence, with human oversight remaining crucial.