WeiPipe: Weight Pipeline Parallelism for Communication-Effective Long-Context Large Model Training
To appear on PPoPP '25. A novel pipeline parallelism that communicate model weight rather than activation under long-sequence scenarios.
Nov 10, 2024
Concerto: Automatic Communication Optimization and Scheduling for Large-Scale Deep Learning
A compiler framework designed to address these challenges by automatically optimizing and scheduling communication
Oct 27, 2024
HeteGen: Efficient Heterogeneous Parallel Inference for Large Language Models on Resource-Constrained Devices
Efficient Heterogeneous Parallel Inference System for LLM on resource-constrained devices.
May 13, 2024
DSP: Dynamic Sequence Parallelism for Multi-Dimensional Transformers
Dynamic Sequence Parallelism for multi-dimensional transformers.
Jan 1, 2024
Hanayo: Harnessing Wave-like Pipeline Parallelism for Enhanced Large Model Training Efficiency
Accepted by SC '23. Efficient Pipeline Parallelism System for LLM.
Nov 11, 2023