2017年度人工智能指数报告（英文版）_99页_4mb

AI Index 2017 Annual Report Summary

Core Content

The AI Index 2017 Annual Report is an open, not-for-profit project initiated by the One Hundred Year Study on AI at Stanford University (Al100). Its purpose is to track AI activity and progress using data, aiming to facilitate informed discussions about the field. The report provides a comprehensive overview of AI trends across academia, industry, and public interest, highlighting both achievements and limitations in measuring AI progress.

Main Sections and Key Findings

Volume of Activity

• Academia:

  • The number of AI-related papers published annually in the Scopus database has increased by over 9x since 1996.
  • Course enrollment in introductory AI and ML courses at Stanford has grown 11x since 1996, indicating rising academic interest in AI.

• Industry:

  • The number of active US AI startups has increased 14x since 2000.
  • VC funding for AI startups in the US has grown 6x since 2000, showing strong industry investment.

• Robot Imports:

  • Industrial robot shipments to North America and globally have seen significant growth, reflecting increased adoption of automation.

• Open Source Software:

  • GitHub usage for AI and ML software (e.g., TensorFlow and Scikit-Learn) indicates developer interest and usage. The number of stars for these packages has increased, showing growing popularity.

Technical Performance

• Vision:

  • Object Detection error rates in the ImageNet competition dropped from 28.5% in 2010 to below 2.5% in 2016, with human performance at around 5%.
  • Visual Question Answering (VQA) performance has improved, though the VQA 1.0 dataset is being surpassed by VQA 2.0.

• Natural Language Understanding:

  • Parsing performance on the Penn Treebank WSJ dataset shows AI systems are getting closer to human-level accuracy.

• Machine Translation:

  • AI systems have achieved human-level performance in translating news between English and German, as seen in the WMT Competition.

• Question Answering:

  • SQuAD v1.1 benchmark shows AI systems are capable of answering questions from text, with performance approaching human levels.

• Speech Recognition:

  • In the Switchboard HUB5'00 dataset, Microsoft and IBM have achieved human-parity performance in speech recognition.

• Theorem Proving:

  • AI systems have improved in solving mathematical problems, though tractability (ability to solve problems) remains a challenge.

• SAT Solving:

  • SAT solvers have shown significant improvements in solving industry-applicable problems.

Derivative Measures

• Academia-Industry Dynamics:

  • The report explores the relationship between academic and industry activity, normalizing data from 2000 to compare growth.
  • Initially, academic activity (papers and enrollment) drove progress, but industry investment became the main driver after 2010.

• AI Vibrancy Index:

  • This index combines academic publishing, course enrollment, and VC investment to quantify the liveliness of the AI field.
  • The index shows a steady increase in AI activity across all sectors.

Towards Human-Level Performance

• The report outlines several notable milestones where AI systems have matched or exceeded human performance:
  • Othello (1989): Program BILLL beat top human players.
  • Checkers (1995): Chinook beat the world champion.
  • Chess (1997): IBM's DeepBlue defeated Gary Kasparov.
  • Jeopardy! (2011): IBM Watson won the show.
  • Atari Games (2015): DeepMind's system achieved human-level performance in many games.
  • Go (2016): AlphaGo beat Lee Sedol 4-1.
  • Skin Cancer Classification (2017): AI system matched dermatologists in diagnosing skin cancer.
  • Speech Recognition (2017): Microsoft and IBM achieved human-level performance.
  • Poker (2017): Programs like Libratus and DeepStack outperformed human professionals.
  • Pac-Man (2017): AI system reached maximum score in the game.

What's Missing?

• Technical Areas:

  • The report lacks coverage of areas like dialogue systems, planning, and continuous control in robotics due to a lack of standardized benchmarks.
  • Commonsense reasoning is underrepresented, as progress in this area is hard to measure.

• International Coverage:

  • The report is US-centric, missing significant AI activity in countries like China.

• Diversity & Inclusion:

  • The report highlights the need to measure who participates in AI discussions and who influences AI research and deployment.

• Government and Corporate Investment:

  • VC investment data is limited to the US and does not include government and corporate R&D spending globally.

• Impact in Specific Verticals:

  • The report acknowledges the need to measure AI's impact in specific industries and applications.

Conclusion

The AI Index 2017 Annual Report provides a snapshot of AI's growth and progress, emphasizing the increased activity in both academia and industry. It introduces the AI Vibrancy Index as a new metric to assess the field's vitality. However, the report also acknowledges several limitations, including lack of international data, standardized benchmarks, and diversity metrics, and invites further community involvement to improve future reports.