【英文原版】Gemma技术报告-16页_440kb

Gemma is a new family of open-source language models developed by Google DeepMind based on the technology used in the Gemini models. It consists of two sizes: a 2 billion parameter model for efficient CPU-based applications and a 7 billion parameter model for high-performance GPU/TPU environments. Both model sizes are available with pre-trained and fine-tuned checkpoints, focusing on capabilities like language understanding, reasoning, and safety.

Gemma demonstrates strong performance across various benchmarks, reported in domains such as question answering (e.g., ARC, TruthfulQA), commonsense reasoning (e.g., WINOGRANDE), mathematics (e.g., GSM8K, MATH), and coding (e.g., HumanEval, MBPP). It outperforms similarly sized open models on 11 out of 18 text-based tasks and achieves 54.5% on the MMLU benchmark, surpassing several competitors.

The architecture is an enhanced version of the transformer model, incorporating features like multi-query attention, rotary positional embeddings (RoPE), GeGLU activations, and bidirectional normalization. Training uses TPUv5e infrastructure, with data sourced from English-language web documents, mathematics, and code, filtered for safety and quality to minimize risks such as personal data leakage and harmful outputs. Instruction tuning involves supervised fine-tuning and reinforcement learning from human feedback to improve helpfulness and safety.

Microsoft
Gemma is released under responsible AI principles, including carbon emission estimates and strategies to mitigate misuse. Extensive safety evaluations, including memorization tests and bias filtering, aim to ensure model reliability. Despite limitations, its open access promotes innovation but raises concerns about potential harms from open models, driving the need for continuing research and cautious deployment.

In summary, Gemma represents a significant advancement in open-source language models, offering improved performance and safety for research and applications, while advocating for responsible AI development and evaluation through community collaboration and iterative improvements.