多模态大语言模型在文本丰富图像理解中的应用：全面综述_17页_2mb

Summary of "Multimodal Large Language Models for Text-rich Image Understanding: A Comprehensive Review"

Core Content

This paper presents a comprehensive review of Multimodal Large Language Models (MLLMs) for Text-rich Image Understanding (TIU). It outlines the evolution of TIU MLLMs, discusses their architectures, training pipelines, datasets, and benchmark performances, and highlights the challenges and promising directions in the field.

Main Points

1. Text-rich Image Understanding (TIU)

• TIU involves both perception (e.g., text detection, recognition, formula and document layout analysis) and understanding (e.g., key information extraction, document-based visual question answering).
• Traditional approaches used specialized models or multi-stage pipelines, but recent advances have integrated these tasks into end-to-end models using vision-language paradigms.

2. Development Timeline

• Pre-LLM Era (2019-2022): Models like LayoutLM and Donut used modality-specific pre-training objectives and OCR-derived supervision.
• Post-LLM Era (2023-present): MLLMs integrate LLMs with visual encoders and use unified attention mechanisms for joint processing of visual and linguistic elements.

3. Model Architecture

• TIU MLLMs generally consist of three components: Visual Encoder, Modality Connector, and LLM Decoder.
• Visual Encoder: Can be OCR-free (e.g., CLIP, ConvNeXt, SAM, DINOv2, Swin-T, InternViT), OCR-based (e.g., LayoutLMv3), or a mixture of encoders (e.g., CLIP + SAM, CLIP + LayoutLMv3).
• Modality Connector: Aligns visual features with language features. Methods include:
  • Token Compression: e.g., Pixel Shuffle
  • Token Reduction: e.g., Cross Attention, H-Reducer, C/D-Abstractor, Attention Pooling
• LLM Decoder: Utilizes powerful language models like LLaMA, Qwen, Vicuna, and InternLM for semantic reasoning and final answer generation.

4. Training Pipeline

MLLMs for TIU are trained through three stages:

• Modality Alignment (MA): Focuses on aligning visual and language features using tasks like reading full text, reading partial text, and parsing (charts, tables, formulas).
• Instruction Alignment (IA): Enhances the model's ability to understand and answer questions using structured instruction templates. It includes:
  • Level 1: Visual-Semantic Anchoring (Answer within or without the image)
  • Level 2: Prompt Diversity Augmentation (Rephrasing questions to improve robustness)
  • Level 3: Zero-shot Generalization (Using reasoning strategies like Chain of Thought (CoT))
• Preference Alignment (PA): Aims to align model outputs with human preferences through reinforcement learning or reward modeling.

5. Datasets and Benchmarks

• Common benchmarks for TIU include DocVQA, InfoVQA, ChartQA, and TextVQA.
• Performance metrics are reported on these benchmarks for various models.

6. Challenges and Trends

• Challenges: Limited adaptability to open-domain scenarios, inefficiency with long sequences, and difficulty in parsing complex document elements like tables and formulas.
• Trends: The integration of hybrid encoder architectures, token compression/reduction techniques, and zero-shot reasoning is becoming more prevalent. There is also a growing emphasis on multilingual support and robust instruction handling.

Key Models and Performance

Conclusion

This survey provides a structured overview of TIU MLLMs, emphasizing the importance of modality alignment, instruction alignment, and preference alignment in their development. It highlights the transition from specialized models to unified, end-to-end MLLMs, which offer greater flexibility and generalization. The paper also identifies key challenges and suggests future research directions, including more efficient token processing, better parsing of complex document elements, and improved zero-shot reasoning capabilities.