WeiPipe: Weight Pipeline Parallelism for Communication-Effective Long-Context Large Model Training
To appear on PPoPP '25. A novel pipeline parallelism that communicate model weight rather than activation under long-sequence scenarios.
Nov 10, 2024
Hanayo: Harnessing Wave-like Pipeline Parallelism for Enhanced Large Model Training Efficiency
Accepted by SC '23. Efficient Pipeline Parallelism System for LLM.
Nov 11, 2023