Sequence Parallelism

WallFacer: Guiding Transformer Model Training Out of the Long-Context Dark Forest with N-body Problem

WallFacer: Guiding Transformer Model Training Out of the Long-Context Dark Forest with N-body Problem

Efficient Sequence Parallelism System for Transformer model training.

Jun 30, 2024

DSP: Dynamic Sequence Parallelism for Multi-Dimensional Transformers

Dynamic Sequence Parallelism for multi-dimensional transformers.

Jan 1, 2024