会议3_利用现代_GPU_集群基于_GPU_的高效压缩方案加速_MPI_ALLREDUCE_通信_21页_2mb

Summary of Accelerating MPI AllReduce Communication with GPU-Based Compression

1. Introduction & Motivation

The AllReduce operation is a critical collective communication in high-performance computing (HPC) and distributed deep learning (DL) training. Existing algorithms suffer from performance issues due to limited network bandwidth. The study proposes two design approaches to integrate GPU-based compression at the collective level, addressing the limitations of point-to-point compression by enabling overlap with send/receive operations and reducing redundant computations.

2. Design Approaches

Ring AllReduce with Collective-Level Online Compression

• Implements compression on individual data chunks during ring communication.
• Aggregates values in a ring fashion, applies compression/decompression, and transfers data, achieving overlap between computation and communication.

Recursive-Doubling AllReduce with Collective-Level Online Compression

• Utilizes specific process pairs for data exchange, compressing entire GPU data in fewer steps.
• Reduces the number of data transfers and optimizes kernel execution across multiple CUDA streams, enabling efficient overlap with send/receive operations.

Both approaches optimize the ZFP compression library for execution in multiple streams and integrate with the MVAPICH2 MPI implementation.

3. Performance Evaluation

Benchmark-Level Evaluation

• Latency Reduction: Showed significant improvements in MPI AllReduce latency across various GPUs and scales:
  • Ring-based compression reduced latency by up to 85.3% on Pitzer (V100 GPUs) and up to 80.1% on MRI (A100 GPUs) and Frontera (RTX5000 GPUs).
  • Recursive-Doubling compression reduced latency by up to 75.5% and 85.0% on Pitzer and Frontera, respectively.
• Comparison with point-to-point compression demonstrated better latency reduction (e.g., Ring+ZFP showed up to 66.1% reduction over P2P compression).

Application-Level Evaluation

• Deep Learning Training: Applied to PyTorch's Distributed Data Parallel (DDP) training of DNN models (e.g., Wide_ResNet50_2, ResNeXt101-32x8d, ConvNeXt_Base).
  • Reduced training time by up to 35.7% with Ring-based compression and up to 32.3% with Recursive-Doubling, using CIFAR10 dataset.
  • Maintained similar training accuracy with high compression rates, but accuracy dropped with lower rates due to compression errors.

4. Conclusion

The proposed collective-level online GPU compression schemes for Ring and Recursive-Doubling AllReduce algorithms significantly improve communication efficiency on modern GPU clusters. Benchmarks show up to 85.3% and 75.5% latency reduction compared to baseline, and application tests demonstrate up to 35.7% training time reduction in DDP training. Future work focuses on extending compression to other parallel strategies for larger DL models.