大语言模型的后训练_深入探究推理_英_31页_2mb

Summary of LLM Post-Training Methods

Fine-Tuning in LLMs:

• Fine-tuning adapts pre-trained LLMs to specific tasks or domains using supervised data.
• Common issues include overfitting and high computational costs. Parameter-efficient methods like LoRA and adapters address these.
• Techniques balance specificity (task performance improvements) with versatility (generalization), though trade-offs exist.

Reinforcement Learning (RL) for LLMs:

• RL enhances alignment and reasoning by optimizing policies with human/ai feedback.
• Methods like PPO, DPO, and ORPO use preference-based rewards for tasks requiring coherence, safety, and multi-step reasoning.
• Challenges include sparse rewards, feedback hacking, and reward model drift, addressed through adaptive strategies and credit assignment mechanisms.

Scaling and Efficiency:

• Scaling involves increasing model size and computational resources for improved performance and efficiency.
• Techniques like LoRA, adapters, and knowledge distillation optimize inference and training, allowing smaller models to retain capabilities.
• Efficient inference strategies include pruning, quantization, and MRC, balancing cost and performance.

Test-Time Scaling (TTS):

• TTS enhances reasoning at inference time without modifying the model, using methods like beam search, best-of-N sampling, and sequential refinement.
• Techniques handle variable computational budgets, dynamically allocating resources for easy/trivial tasks versus hard complex ones.
• Decoder strategies like chain-of-thought prompting improve reasoning by breaking down problems into smaller steps.

Overall Contributions:

• This survey provides a taxonomy of post-training techniques, covering fine-tuning, RL, scaling, and test-time scaling.
• It offers practical guidance through benchmarks and datasets, addressing challenges in RL alignment, data scaling, and personalization.
• Pioneering directions include hybrid methods, like combining RL-based tuning with test-time scaling and advanced MRC frameworks.