PipeDream: Efficient DNN Training Via Pipeline Parallelism.pdf

PipeDream is a system that enhances the efficiency of Deep Neural Network (DNN) training by combining intra-batch parallelism with inter-batch pipelining. Key aspects include:

- Addressing Communication Overhead: PipeDream reduces inter-worker communication by limiting it to consecutive layers assigned to different workers, which are often smaller than the entire model. This results in peer-to-peer communication instead of all-to-all.

- Handling Bi-directional Training: Unlike traditional pipelining, DNN training involves a forward pass followed by a backward pass through the same layers. PipeDream's 1F1B scheduling algorithm ensures high hardware utilization by alternating between forward and backward passes, with no pipeline stalls or flushes in steady state.

- Maintaining Statistical Efficiency: PipeDream uses different versions of model weights to maintain statistical efficiency comparable to data parallelism. Each backward pass results in weight updates, and the next forward pass uses the latest version of weights available.

- Automatic Layer Partitioning: PipeDream automatically partitions DNN layers among workers to balance work and minimize communication.

- Performance: Extensive experimentation shows that PipeDream trains models up to 5.3× faster than commonly used intra-batch parallelism techniques, while achieving high accuracy.